I have two time series that give me a monthly count. One is a reference series that is meant to be used as a control; the other has a 'treatment' or program applied to it mid-way through. Both are expected to trend upward broadly.
I have been asked to identify if the treatment/program is having a statistically significant impact on the resulting monthly counts. Preferably with a simple and intuitive test.
It has been suggested that I should:
- Transform the counts of both series to month-over-month percentage change.
- Perform a Man-Whiney-U (MWU) test on the month-over-month change series pre-treatment to test if the series come from the same distribution and then do so again post-treatment.
However, I'm not sure about this approach and here is my reasoning:
- Based on this cross validated question I believe that the MWU test requires both independence within each sample, as well as between samples. Since the samples are time-series, it seems unlikely that I have independence within each sample (even though the month-over-month percentage transformations make them look like white-noise). Furthermore, since one series is meant to act as a reference for the other, it also seems unlikely that I have independence between samples.
But, I am not confident this reasoning is correct and do not have experience in this area. Equally, am unsure of how to illustrate with examples or synthetic data how the suggested approach could be problematic or lead to making bad inferences. Can anyone enlighten or assist?
I am also open to suggestions on how to better approach this problem, but based on this question I want to recommend we try a Chow test on the pre-vs-post treatment timeseries (without the month-over-month transformation, or maybe with a compound monthly growth transformation) as a better approach.
