3

I would like to calculate whether the result of my survey is significant or not. There are two designed interfaces and in the survey, I have asked one question because for example if an interface A is liked 23 over B among 26 people, then interface B is liked 3 over A among 26 people. Therefore I kept the questions just one.

I tried to perform chi-square test but whenever I try to apply the formula, I am stuck because all examples I see require at least 2*2 matrix.

New Interface
Liked 23
Disliked 3

I tried to use the chi-square formula, but I can't do it because of the reason I explained above. I have calculated my conversion rate for the Like as %88.4 and Disliked as %11.5. But I can't do any further since I can't apply an example similar to my situation. I tried to perform the formula below for the chi-square but I got stuck.

enter image description here

I was wondering if anyone could help me with this? Thanks in advance!

Richard Hardy
  • 67,272
calvinjam
  • 33
  • 1
    Just to say explicitly what is implicit in the answers: Any question about statistical significance only makes sense relative to a specified null hypothesis and alternative. A result is not in itself "significant" but may provide significant evidence against a null hypothesis. The answers assume that your null hypothesis is that the true probabilities for like and dislike are the same, but it would have been your job to specify this, and in principle it could be different. – Christian Hennig Jan 09 '22 at 12:23
  • 1
    In fact in the given situation I think data are better presented using a confidence interval for the probability rather than a test; I don't think any point null hypothesis is particularly interesting. (There's much discussion in statistics and science these days about the overuse and abuse of significance tests.) – Christian Hennig Jan 09 '22 at 12:25

2 Answers2

7

Chi square test requires you have a prior notion of what is "expected". Under the assumption there is no difference in interfaces, you would expect an equal proportion of people would like and dislike the interface.

Therefore, $E=13$ is the expected number of people who would like the interface (your entire sample multiplied by the expected proportion who would like it). This is also the expected number of people who would dislike it.

The $X^2$ statistic is then

$$ \dfrac{(23-13)^2}{13} + \dfrac{(3-13)^2}{13} = \dfrac{2}{13}100 \approx 15.3 $$

This test should have one degree of freedom, so we reject the null hypothesis that equal proportions of people like and dislike the interface with a p value far below 0.001.

Demetri Pananos
  • 36,121
  • 1
    Thank you so much!

    This is actually the same result that I found yesterday but I wasn't sure if taking 13 was correct or not, thanks for clarifying that. Since I am quite new to the topic, by looking at the Probability of Exceeding the Critical Value table with a degree of freedom 1, my p number is below 0.001 and therefore can say it's less than 3.84 for 0.05 hence is statistically significant? Would this be a correct statement looking at this result?

     – calvinjam Jan 08 '22 at 15:36
  • @calvinjam That would be correct. – Demetri Pananos Jan 08 '22 at 15:41
  • Thank you so much for your help I really appreciate it! – calvinjam Jan 08 '22 at 15:45
  • 2
    +1. The chi-squared p-value, which is an approximation, is remarkably close to the correct p-value given by the Binomial distribution. – whuber Jan 08 '22 at 19:12
  • 1
    Why is there 1 degree of freedom if it's a 2x1 matrix? Surely it should be (c-1)*(r-1), where c and r are the columns and rows resp.? @DemetriPananos – Jay Ekosanmi Jan 09 '22 at 12:43
  • 2
    @JayEkosanmi See "Chi-square goodness of fit test". When testing the frequency of categories as we are here, the degrees of freedom are $C-1$ where $C$ is the number of categories. Here $C=2$. Martin Bland's "Introduction to Medical Statistics" has more. – Demetri Pananos Jan 09 '22 at 15:21
  • Thank you so much for the clarification for the degree of freedom. I'm really sorry but I'm still confused. I look at my chi square results which is 15.3 then I check from the table where the α = 0.05 and for degree of freedom is 1, and my number is larger than 3.84, does this mean both hypotheses are rejected and my results are not significant? I have just checked the "Chi-square goodness of fit test". @DemetriPananos – calvinjam Jan 09 '22 at 15:54
  • 1
    @calvinjam Your test statistic is larger than the 3.84 so you would reject your null hypothesis – Demetri Pananos Jan 09 '22 at 16:06
  • Does this mean I also reject my alternative hypotheses ? Also my p value is very small and below 3.84 and is that why the test significant ? @DemetriPananos – calvinjam Jan 09 '22 at 16:22
  • 1
    @calvinjam You only reject the null. Because the test statistic is greater than the critical value (3.84) you conclude the result is statistically significant. – Demetri Pananos Jan 09 '22 at 16:26
  • Oh now I understand everything. Thank you so much for your effort and make everything clear for me. – calvinjam Jan 09 '22 at 16:27
7

A chi-squared test is OK because $n = 26$ is large enough for the chi-squared statistic to have approximately the distribution $\mathsf{Chisq}(\nu=1),$ giving the P-value $0.00009 < 0.001 = .1\%.$ (See @whuber's Comment.)

 1 - pchisq(15.3, 1)
 [1] 9.171651e-05

Here is an exact binomial test in R of $H_0: p_A = .5$ against $H_a: p_A \ne .5,$ where $p_A$ is the population proportion favoring A. [Unless another null value is stated, binom.test assumes $H_0: p_A = 0.5.]$

binom.test(23, 26)

    Exact binomial test




data:  23 and 26
number of successes = 23, number of trials = 26, p-value = 8.798e-05
alternative hypothesis: 
 true probability of success is not equal to 0.5
95 percent confidence interval:
 0.6984596 0.9755419
sample estimates:
probability of success 
             0.8846154

The exact P-value can be computed as $$P(X \le 3)+P(X \ge 23)=8.797646e-05\approx 0.000087976,$$ where $X\sim\mathsf{Binom}(26,0.5).$

sum(dbinom(c(0:3,23:26), 26, .5))
[1] 8.797646e-05

Note: If you had suspected, before seeing data, that A would be favored over B, then you might have used a one-sided binomial test of $H_0: p_A = .5$ against $H_a: p_A > .5,$ For that test, the P-value (half as large as above) would be computed by looking only in the right tail.

binom.test(23, 26, p=.5, alt="greater")

    Exact binomial test




data:  23 and 26
number of successes = 23, number of trials = 26, p-value = 4.399e-05
alternative hypothesis: 
 true probability of success is greater than 0.5
95 percent confidence interval:
 0.728098 1.000000                # one-sided CI
sample estimates:
probability of success 
             0.8846154

In effect, the chi-squared test is inherently two-sided--on account of the squaring.

BruceET
  • 56,185
  • 2
    Thank you so much for a great answer. So the result 15.3 and p-value from binom.test can both be used to determine whether a test result is significant or not is that correct? For 15.3 I check for the 0.05 from Probability of Exceeding the Critical Value and the number is greater than 3.84, then I say that the test result is significant. On the other hand I see that p-value is 0.000087 but from there can I reach if the results are significant or not ? Do I use the table for the p value as well ? I'm sorry I really have little knowledge for the topic and I'm trying to comprehend everything. – calvinjam Jan 08 '22 at 23:32
  • 1
    To match results in a parallel fashion, I'd say (approx.) chi-squared statistic 15.3 yields P-value 9.171651e-05, leads to rejecting the null hypothesis that $p_A = 0.5.$ (You could use a printed chi-sq table to see that 15.3 is significant below 1% level. Also , $X=23$ preferring A yields (exact) P-value $8.798e-05$ leads to rejecting the same null hypothesis. You could do an exact computation with a binomial distribution or a normal approximation to see that the exact binomial test leads to rejection. // The binomial test could be on- or two-sided, but in effect the chi-sq test is 2-sided. – BruceET Jan 09 '22 at 00:17
  • 2
    Most statistical software programs (including R) show P-values from internal computations so that it is not necessary to use chi-squared or normal tables. // If the appropriate P-value is less than 1%, then the corresponding null hypothesis is rejected at the 1% level. // Try following several examples in a statistics text, preferably ones that include computer results. – BruceET Jan 09 '22 at 00:17