6

I work on an ecommerce site, and I'm trying to determine a feature's impact on conversion.

This is not an AB test; rather, 100% of customers had access to the feature- whether or not they used the feature is indicated in "Segment" below.

In the past 12 months:

Segment Converted Didn't convert Total
Used feature 629 20,570 21,097
Didn't use feature 243,273 8,131,398 8,374,671

Plain English: Of 21,097 customers who used feature, 629 or 2.98% converted. Of 8,374,671 who didn't use feature, 243,273 or 2.90% converted.

At first glance "Used feature" has higher conversion, but I'm wondering what other calculations I need to make a statistically sound conclusion.

New to stats.stackexchange so apologies if formatting is improper!

michaelscarn
  • 63

3 Answers3

15

You can do the chi-squared test and get a p-value. (I haven't done the test, so I don't know what the p-value is.)

What you cannot do — without making a big assumption — is to claim that the feature increases the probability of conversion. It may be the case that users who were more likely to convert were also more likely to use the feature. In other words, the population of customers who use and don't use the feature may not the same.

This conundrum illustrates why A/B tests are useful: by randomizing customers to use the feature or not, we make sure that those who have access to the feature are comparable (exchangeable) with those who do not and so we can draw conclusions about the feature's effectiveness to increase conversion.

dipetkov
  • 9,805
6

If all the customers are independent (e.g. you don't have data on the same customer multiple encounters) then this is the classic 2 by 2 table that can be analyzed using a Chi-Squared test: https://en.wikipedia.org/wiki/Chi-squared_test.

Greg Snow
  • 51,722
1

Normally, testing involving two samples requires more complicated analysis than testing with one sample. But in this case, the sizes are so unbalanced that it's a reasonable approximation to just do an analysis on the smaller sample. We can take a null hypothesis that the smaller sample is from a Poisson distribution whose $\lambda$ is equal to the percentage of the large sample times the size of the small sample. That gives $0.029*20570=596.53$. This gives a p-value of 8.9%, which is not significant. You could also use a binomial distribution, which gives a p-value of 8.6%.

Acccumulation
  • 5,206