3

I am examining the hypothesis that the average conversion rate of users who click on a particular feature is significantly different from that of users who do not click on the feature. For my analysis, which would be more suitable - a test comparing group means, or a chi-square test, comparing group proportions? And why?

Thalassophile
  • 2,895
Vaishali Venkatesh Prasad
  • 31
  • 1
    Conversion seems to be a binary ouctome (yes/no), so you should go via prop.test(). Note that prop.test() is equivalent to a Chi-squared test comparing four cells (conversion + click, no conversion + click, conversion + no click, no converion + no click), but cannot handle larger matrices. – stefgehrig May 29 '20 at 07:50
  • Thanks! I read also that the hypothesis for chisquare tests like prop.test() is that there is no significant relationship between the variables, but for the t.test(), it is there is no significant difference in means between the two groups. In my case, my hypothesis falls under the latter category - does a prop.test() still apply? – Vaishali Venkatesh Prasad May 29 '20 at 10:53
  • the hypothesis of the chi square test can be refrased to be "the target proportion doesn't change between categories". this actually corresponds to the definition of indipendence between random factors. – carlo May 29 '20 at 13:12
  • The two-sample t-test of proportions and two-group chi square are equivalent. https://stats.stackexchange.com/questions/173415/at-what-level-is-a-chi2-test-mathematically-identical-to-a-z-test-of-propo – Estimate the estimators Feb 04 '23 at 06:28

3 Answers3

1

In terms of two average conversion rates, I think you are comparing two categorical variables, and hence two proportions other than two means, then I think you should utilize the chi-square test.

Lerner Zhang
  • 6,636
  • 1
  • 41
  • 75
1

An appealing aspect of the t-test is that it is pretty robust to deviations from the normality ideal. Consequently, the t-test might be a reasonable idea when you don’t know the distribution.

However, binary outcomes are so simple that they must have a binomial distribution. Consequently, we can use a test that is specifically designed for binomial distributions. Thus, the various types of proportion testing, such as chi-squared, are likely to give superior performance.

Galen
  • 8,442
Dave
  • 62,186
0

Assuming you are analyzing binary outcomes (and not the estimates of rates for each users say by counting up the number of times they clicked and dividing by the number of times they could have clicked) then both the z test and the chi-square test will yield approximately the same result, with differences becoming smaller as the size of the data grows large. See my answer here.

a test comparing group means, or a chi-square test, comparing group proportions? And why?

These are all equivalent. The group means are the group proportions, and as I've mentioned above the chi-square test is equivalent to the test of proportions.

In particular, a z test is best in my opinion because you can get uncertainty on whatever estimate you are making (e.g. risk difference, risk ratio, odds ratios, etc). A t test is not neccesary because the estimated variance is a function of the estimated mean. You could do a t test, and you'd probably get the same result, but you don't need to.

Demetri Pananos
  • 36,121