0

First up, apologies that this probably a dumb question.

I am switching careers and just getting started in trying to understand statistical significance in A/B testing. I've come into issues quickly in identifying the right statistical test.

I am trying to calculate statistical significance across a variety of KPIs within an A/B test for a mobile game (with an 80:20 control:experiment sampling ratio), but am struggling to identify which tests fit which KPIs.

For instance: comparing ARPU (average revenue per user) seems to require a different method to comparing average revenue per user per game. But I'm not successfully grasping the difference and why (binomial vs non-binomial? parametric vs non-parametric?).

I'm ultimately seeking some guidance on which tests would be appropriate for these KPIs, or guidance as to how I could determine which tests would be appropriate. For each of the below I have already gathered metrics but now want to validate the significance of the results.

  1. active days per user
  2. active days per user out of possible days
  3. games played per user
  4. games played per user per active day
  5. average revenue per user
  6. average revenue per paying user
  7. average revenue per daily active user
  8. average revenue per game played
  9. paid conversion rate (chi squared?)
  10. average days from install to paid conversion
  11. D1 retention rate, D3 retention rate, D7 retention rate, D14 retention rate

Any help, big or small, would be incredibly valuable. Thank you.

Ned Miles
  • 101

0 Answers0