First up, apologies that this probably a dumb question.
I am switching careers and just getting started in trying to understand statistical significance in A/B testing. I've come into issues quickly in identifying the right statistical test.
I am trying to calculate statistical significance across a variety of KPIs within an A/B test for a mobile game (with an 80:20 control:experiment sampling ratio), but am struggling to identify which tests fit which KPIs.
For instance: comparing ARPU (average revenue per user) seems to require a different method to comparing average revenue per user per game. But I'm not successfully grasping the difference and why (binomial vs non-binomial? parametric vs non-parametric?).
I'm ultimately seeking some guidance on which tests would be appropriate for these KPIs, or guidance as to how I could determine which tests would be appropriate. For each of the below I have already gathered metrics but now want to validate the significance of the results.
- active days per user
- active days per user out of possible days
- games played per user
- games played per user per active day
- average revenue per user
- average revenue per paying user
- average revenue per daily active user
- average revenue per game played
- paid conversion rate (chi squared?)
- average days from install to paid conversion
- D1 retention rate, D3 retention rate, D7 retention rate, D14 retention rate
Any help, big or small, would be incredibly valuable. Thank you.