5

The goal is to find out if the frequency difference of a word "X" in two texts/corpora is significant. We have a $2\times 2$ contingency table of observed and expected frequencies.

To find out if the frequency difference is significant, I use the chi-square test. But which one exactly? Chi-Square Test of Independence?

  • $\alpha = 0.05$
  • one degree of freedom
  • critical value of the chi-square distribution -> 3.84

The formulated hypotheses:

null hypothesis: The frequency of the word "X" in text/corpus A differs not significantly from the frequency of the word "X" in text/corpus B

alternative hypothesis: The frequency of the word "X" in text/corpus A is significantly higher than the frequency of the word "X" in text/corpus B

  • Observed Frequency of word X in Corpus A: 2901
  • Observed Frequency of word X in Corpus B: 3019
  • Observed Frequency of all other words in Corpus A: 90381
  • Observed Frequency of all other words in Corpus B: 80281
  • Total words of Corpus A: 93282
  • Total words of Corpus B: 83300

Result: The chi-square statistic is 35.9258. This value is greater than the critical value of 3.84, accordingly the result is significant. The calculation of the p-value is not mandatory, is it? Suffice it to say that the test statistic of 35.9258 is higher than the critical value of 3.84 of the chi-square distribution?

Back to our result: The word "X" occurs significantly more often in book B and not, as assumed in the alternative hypothesis, in book A. How would you formulate the hypothesis evaluation? It is obvious that we reject the null hypothesis because the frequency difference is significant, but what do we do with our alternative hypothesis?

yahiro
  • 97
Sunkey
  • 61

2 Answers2

7

To your first question, a chi-square goodness of fit test only tests if the frequency distribution is different from your expectation. In this case, it is only testing one categorical factor. A chi-square test of independence tries to test if there is a relationship between multiple categorical factors. So in your case, you would be using the chi-square test of independence since you are using a 2x2 contingency table. That distinction can be seen in this useful link.

A chi-square test does not test the alternative hypothesis. It only tests the null hypothesis, which is that there is no relationship between your factors. This subject can be a bit of a rabbit hole, but for the sake of a chi-square test, just know that rejecting the null does not mean you have proven the alternative hypothesis.

The p value is important to a degree to have (you often must report it in academic journals as a requirement to their manuscript rules), but I think that you would also want to obtain an effect size to approximate the magnitude of the effect, such as Yule's Q coefficient. You should also meet the general assumptions required for a chi-square test as well.

Shawn Hemelstrand
  • 13,543
4

It appears that you have not formulated your null and alternative hypotheses in a way that is consistent with what you want to test. Your null hypothesis should be that the frequency of the word in corpus B is greater than or equal to the frequency of the word in corpus A.

Given that, once we see that the observed frequency of the word in corpus B is, in fact, greater than in corpus A, we already know that we aren't going to reject the null hypothesis.

An appropriate test for this one-sided hypothesis would be based on comparing the observed frequencies. In the real world, words aren't independent of each other, so any test based on the independence assumption will be seriously wrong. Still, for illustrative purposes, we carry on. In this case, given the large sample size and our assumption of independence, a Normal approximation for the distribution of the observed frequencies will work well.

Our test statistic $\tau$ is:

$$\tau = {\hat{p}_A-\hat{p}_B \over \sqrt{\hat{p}_0(1-\hat{p}_0) \cdot \left({1 \over N_A}+{1\over N_b}\right)}}$$

where $\hat{p}_A, \hat{p}_B$ are the observed percentages of the time the word occurs in corpus A and B respectively, and $\hat{p}_0$ is the observed percentage in the combined sample.

Our null hypothesis is that $\hat{p}_A \leq \hat{p}_B$. The calculations result in $\tau = -5.99$. The associated p-value is obviously close enough to zero that it's not worth calculating. Consequently, we fail to reject the null hypothesis.

jbowman
  • 38,614