0

I've spent several hours online researching an appropriate means of using statistics to determine whether or not variation test results are meaningful, or just chance.

If the testing took the format of an email campaign, where two different sequences of emails were sent to a population of people, what would be an appropriate means of determining success?

As of and up to September 1st 2014, new email subscribers were sent 5 emails within 30 days. A new email campaign was conceived, where 5 emails were sent but with different content. From September 1st, as new subscribers signed up, each alternating address was placed into either the original sequence of emails or the new sequence.

The goal is that after 30 days the subscribers will become customers.

So I now have data that looks like this:

Group | Subscribers | Customers
A     | 500         | 50
B     | 500         | 60

So the conversion rate for group A is 10% and 12% for group B.

Group B, the new sequence, performs better than group A.

Is there an opportunity to use stats to underline this statement?

I was read lots of posts on Chi Square tests but the thing I couldn't reconcile was "expected". The web is full of examples of voting intentions of men Vs. women in terms of right Vs. left (Republicans and Democrats). In those examples the expected for Null hypothesis was 50/50.

My data is laid out in a way that looks like chi square examples but, since I have no "expected" I am unable to proceed.

Can anyone point me in the right direction here?

Doug Fir
  • 1,568
  • 1
  • 19
  • 36
  • When you say "sent out to a population" ... do you mean he population about which you wish to make inference? Or do you simply mean they were sent out to some number in each group? If you intend the results to generalize outside those 1000 people (as you would if you wanted to say "B does better" about other people), they're not the target population but presumably some kind of sample. – Glen_b Oct 27 '14 at 21:29
  • Assuming that these groups of 500 are intended to be random samples from the population of interest, you'd either use a two-sample proportions test (typically done as a z-test) or a 2x2 chi-square (A/B vs customer/not-customer). Properly organized the two should be equivalent. (There are other possible tests.).... there are many posts on this topic here. – Glen_b Oct 27 '14 at 21:35
  • @Glen_b thanks for the feedback. RE your first comment I guess by population I mean it's not actually a sample: either you were placed in group A or B 50/50 so everyone is included, not just a sample. Presumably that removes inference from the equation since we have the whole population in question? Does this change things? Z-test still apply? The link you provided does not appear to work – Doug Fir Oct 27 '14 at 22:06
  • An example of a two-sample proportions test is here; there are several links describing it here. There's a chi-squared test here but for more than two groups, that shows the subtraction of the count of interest from the population exposed to obtain the second set of counts for the chi-squared test...(ctd) – Glen_b Oct 27 '14 at 22:07
  • (ctd) ...Both tests are mentioned in the table here. $\quad$ Um, I didn't provide a link in my first two comments. It's not clear to me that you do have the whole target population. What use is it to know that something has a higher proportion on some fixed group from the past? What was the purpose of this analysis? – Glen_b Oct 27 '14 at 22:09
  • Thanks again for the feedback I think I'd typed my last comment as you were typing yours. Purpose of test is to know which email sequence to use in the future, so in a sense is a sample in that the whole population are all the people who will sign up in the future but have not yet done so – Doug Fir Oct 27 '14 at 22:26
  • Yes; this is fairly common. You don't have a random sample from the target distribution (because you can't), but you have a group which is hopefully representative. If you allocate to groups A/B at random, you have a randomization argument as far as applying a test for the group effect (and the usual test can be argued to be an approximation to a suitable randomization test); there's still a need to argue that result carries over to the population of interest. Sorry several of my links in my previous couple of comments didn't work as links - you should still be able to copy and paste them – Glen_b Oct 27 '14 at 22:30
  • I was initially thinking this was effectively a duplicate question, but there's enough here now that I think isn't covered by any single question that I should probably write it up as an answer. – Glen_b Oct 27 '14 at 22:32
  • @Glen_b: Can you write it as an answer then? – kjetil b halvorsen Mar 02 '24 at 19:25

0 Answers0