1

My company would like to use control charts (Shewhart, EWMA, or both) to detect changes to the distributions of our key business metrics. For instance, we could sample user signups each day and determine whether a change to our sales webpage has caused the mean to shift. We have two major goals:

  1. To find out quickly when something has changed in a major way (eg. the signup form is broken and mean signups drop 80%) [Shewhart]
  2. To find out if something has caused the distribution to shift in a small way, not more than 2 std devs (eg. changing a button from blue to red caused mean signups to jump up 5%) [EWMA]

The problem is, as far as I can tell, that all control charts assume the ideal mean is constant, when in fact, our user signups are growing steadily independent of any of the intervention we want to track. Rather than just detecting a change, I want to understand if the change was more or less than would be expected from a projection of past user signups.

How can I construct control charts for data where the ideal mean is changing over time?

Travis
  • 111
  • Can you produce forecasts for the number of sign-ups by day, with confidence bounds, then check for sign-ups that occur outside the bound? You can reforecast as well, to keep your confidence bounds tighter. – Michelle Feb 14 '12 at 04:33
  • Thanks Michelle, that approach makes sense to me (very much a layperson), and would be easy to implement. I'm curious if there are weaknesses that get introduced with that approach that aren't present when the ideal mean is fixed. Have you seen that approach documented or studied anywhere? – Travis Feb 14 '12 at 15:11
  • 1
    The main weakness is that, as the forecast goes further into the future, the confidence intervals get too wide to be useful. Also, because forecasts are performed off historical data, they assume the future is like the past, so structural breaks are an issue. Reforecasting should assist in reducing both those issues, although I have had one time series where I could fit a line, but I couldn't predict off it, and it wasn't a random walk. – Michelle Feb 14 '12 at 17:11

0 Answers0