Extracting Revenue Impact from Business Metrics

When I joined System1, I wanted to make an impact as quickly as possible. The first change I pushed into our system was designed to optimize our click-thru rate (CTR) model. I was hoping to see a clear uptick in user engagement, but instead I saw something similar to the graph on this page (Fig. 1).