0

I'm attempting to tie classwork theory with an actual work example just now. Yesterday I attended a lecture on hypothesis testing and my notes and lecture slides cover hypothesis testing with one sample of data.

I want to tie what I'm learning to a real world work question. We are an email marketing vendor and have been asked to split test an email message which is being deployed to the clients database. The goal of the email deployment will be opens. So if we change the subject line can we get more people to open the email?

The results of this experiment might look like this:

                Sent  | Opened | Open Rate
Original Email  9,000 | 900    | 10.0%
New Variant     1,000 | 125    | 12.5%

If my hypothesis is that there is no difference between the emails then what method should I use to get a p-value in order to determine if the difference seen was reasonably likely to happen anyway (>5%)?

  • Chi-Square?
  • Another method?

How should I decide with method to use for the test?

Notes:

  • I'm a learner of stats and at a pretty foundational level right now so please bear that in mind in any responses
  • I'm assuming that the mean I want to compare to is the open rate of those who received the original?
  • I did Google this question before posting. But the search results were crowded out with "Hypothesis test with 1 sample" not "Hypothesis test with 1 observation"
Doug Fir
  • 1,568
  • 1
  • 19
  • 36
  • 1
    Check $\chi^2$ test: http://en.wikipedia.org/wiki/Chi-square_test – Tim Jan 21 '15 at 16:54
  • 3
    You couldn't find much in your searches because you seem to conflate "sample" with "observation." You have a sample with 10,000 observations. You have summarized it with a contingency table. You would like to know whether the responses are independent. All those keywords will lead you directly to the $\chi^2$ test recommended by @Tim. Searching our site will turn up dozens of useful examples and answers. – whuber Jan 21 '15 at 16:59
  • 1
    @whuber yes that seems to be the case: http://tinyurl.com/o2vmfdr. Thanks for the info both – Doug Fir Jan 21 '15 at 18:12
  • Just to follow up and clarify my understanding. We chose a χ2 . At what point would we decide to change this to a t-stat? If we had say 15 variants and not just one? – Doug Fir Jan 21 '15 at 19:58

0 Answers0