Let's assume I have 100s of websites on which I am tracking conversion. At some point I change something to all websites. I want to measure if there is a statistical significant difference between my conversion before and after the change.
My data would look like this:
| Website ID | Views Before | Clicks Before | Conversion Before | Views After | Clicks After | Conversion After |
|---|---|---|---|---|---|---|
| 1 | 23 | 2 | 8.69% | 45 | 4 | 8.88% |
| 2 | 3 | 0 | 0% | 2 | 0 | 0% |
| 3 | 231 | 14 | 6.06% | 123 | 10 | 8.13% |
| 4 | 1220 | 87 | 7.13% | 2435 | 235 | 9.65% |
| 5 | 87 | 1 | 1.15% | 50 | 1 | 2% |
| ... | ... | ... | ... | ... | ... | ... |
My first idea was to use a paired t-test or the wilcoxon signed rank test to compare Converion Before and Conversion After columns. But if I do that, I will loose a lot of information, as a website with 2 views will be seen as equally important as a website with 1000 views. I could also run individual tests on each website, but not sure how to aggregate the result then. Or, I could sum all views and clicks for the Before and After data, but then I'm losing the paired character of my data.
Any ideas about how I should measure this?
