0

What is the most appropriate test to find if the differences in conversion rates are statistically significant in the case I'm comparing multiple different groups? e.g.

I have different groups of ages (teenager, young, adult, elderly) and a conversion rate related to each of them in my website. How can I validate the differences in the conversion rates from a statistical point of view?

class      size    conversion_rate (=num_conversions/size)   
teenager   1000        3%
young       700       3.3%
adult       800       4.7%
elderly     300       2.3%

The ultimate question I'd like to answer is if the trend (e.g., grows until adulthood, then drops) can be attributed to chance or not.

simon
  • 121
  • 1
    Both "validate" and "significance" have technical meanings in statistics that differ from the ordinary meaning of those words. Try rewriting the question in plain english so it is clear what you want to find out. – Harvey Motulsky Oct 19 '22 at 14:20
  • 1
    For example, do you want to know if there are any differences in conversion rates among your age groups? Are there some specific age groups that you would like to compare? Do you want to perform all 6 pairwise comparisons that are possible with 4 age groups? Please edit the question to clarify what you seek, as comments are easy to overlook and can be deleted. – EdM Oct 19 '22 at 16:47
  • Thanks for your comments, I tried to clarify the question – simon Oct 19 '22 at 19:00
  • Could you define what is ‘conversion rate’ and it’s measured? – utobi Oct 19 '22 at 19:05
  • Is your hypothesis about the trend something you had considered prior to the test, or is this only coming about after seeing the trend? – Demetri Pananos Oct 19 '22 at 19:57
  • there's no hypothesis before collecting the data, this is coming after seeing the conversion rates and wondering if any conclusion can be taken – simon Oct 19 '22 at 21:15
  • @simon Any statistical test that is chosen based on what you see in the data is invalid. It's like drawing a target around your arrow. – Christian Hennig Oct 20 '22 at 20:57
  • A test of the null hypothesis "conversion rate is the same for all age groups" doesn't say anything about trends. And knowing the statistical significance for this null hypothesis doesn't seem particularly actionable. What are you going to do if the null hypothesis is rejected and what are you going to do if it's not rejected? – dipetkov Oct 20 '22 at 21:16

2 Answers2

1

If you want to get one numerical value for the statistical significance of the difference between all groups (rejecting the null hypothesis that all proportions are identical), you might want to take a look at the most trivial $\chi^2$ test for the contingency table:

converted not converted
teenager 30 970
young 23 677
adult 38 762
elderly 7 293
Alex
  • 1,057
0

Alex (+1) has the fundamental answer: if you have categorical data the $\chi^2$ test provides a simple test of overall significance. For your data I get a p-value of 0.12, meaning that in 12% of experiments like this you would get at least this large an apparent deviation from independence even if there were no differences among age groups in conversion rates.

Is that far enough from "chance" for your application? It doesn't pass the usual "significance" standard of p < 0.05, but there's nothing magical about that cutoff.

You might do better if you had individual ages corresponding with individual yes/no conversions. Your categorization of a continuous variable like age is generally unwise; it implies unrealistic sudden breaks in behavior at specific ages. Modeling age smoothly as a continuous predctor, for example with a regression spline, could be done in a binomial regression model (e.g., logistic regression). That could be more informative, allowing you to plot estimated conversion rates as a smooth function of age.

If you are stuck with age categories, note that binomial regression will allow you to do analysis among specific categories; the $\chi^2$ test only looks at overall deviations from independence among all categories. For your data, expressed as Alex suggests, you could do the following in R:

glm(cbind(converted,notConverted)~class,data=yourData,family=binomial(link="logit"))

If you pursue this type of work further, look into binomial regression.

EdM
  • 92,183
  • 10
  • 92
  • 267