0

I am looking to run an A/B Test testing the effectiveness of a promotion on revenue but I am concerned about the pre-period skewness of my customer spend. I am splitting my customers into 2 groups of 20K each based on revenue and a few other metrics. After sampling, my means are close to being equal, however I am worried if I can I can use a t-test in this case because almost 30% of my revenue is skewed top 1% of my customer base. My variance is relatively high. Would there be a best test to run?

NSri
  • 13
  • The skewness doesn't make the t-test inappropriate. If you are interested in comparing means, the t-test is fine. The question is whether you should be comparing means. – bdeonovic Feb 09 '23 at 20:49
  • 1
    @bdeonovic Depending on how bad the situation is, the t-test could be rather underpowered compared to the ideal situation of having normal distributions (Frank Harrell has written about this, either here or on his blog), though I agree that more detailed comparisons that just comparisons of means might be useful for this task. – Dave Feb 09 '23 at 21:15
  • @bdeonovic good point, would you recommend comparing something else instead and a good test to use to do so? Potentially the distributions changing? – NSri Feb 09 '23 at 21:53
  • There are lots of posts on this website which address the same or similar question see eg https://stats.stackexchange.com/questions/69898/t-test-on-highly-skewed-data or https://stats.stackexchange.com/questions/110801/should-i-use-t-test-on-highly-skewed-data-scientific-proof-please or https://stats.stackexchange.com/questions/416097/statistical-power-of-t-test-in-mildly-skewed-dataset – bdeonovic Feb 09 '23 at 22:00
  • 1
    among those 3 posts you will find answers by all the heavy hitters on this website including Glen_b, Frank Harrell, and whuber – bdeonovic Feb 09 '23 at 22:02
  • and to directly answer your question you may consider a non-parametric test like Mann-Whitney (https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test) – bdeonovic Feb 09 '23 at 22:05

0 Answers0