Where does p-value come from in McNemar's Test?

Question

I was using this online calculator to work out McNemar's Test on this data set:

enter image description here

I get the results:

The two-tailed P value equals 0.1859

Chi squared equals 1.750 with 1 degrees of freedom.

Now I understand how to calculate the Chi squared but I have no idea where this P value comes from.

Also how does the P value relate to whether there is a significant difference between classifiers?

If someone could explain this like a program in JavaScript or something it would be so much clearer. Every explanation I see throws jargon all over the place.

And do you know what p-values are in general - if not see here or here. — Scortchi - Reinstate Monica, May 17 '14 at 17:23
It isn't clear what you need exactly. Benjamin's answer is good (+1), & you may want to read @Scortchi's links to help you understand what a p-value is in general. It may also help you to get a general overview of McNemar's test, I have a fairly comprehensive one here: What is the difference between McNemar's test & the chi-squared test, & how do you know when to use each? — gung - Reinstate Monica, May 17 '14 at 21:21

bdeonovic · Answer 1 · 2014-05-18T00:37:30.483

7

Asymptotically the McNemar test statistic follows a chi-squared distribution with 1 degree of freedom. So, if $x_{obs}$ is your observed McNemar test statistic, the $p$ value is

$p = \text{Pr}\left\{ \chi^2_1 > x_{obs}\right\}$

but perhaps this is all the jargon and what not that you were saying you were confused about. What the statement above is saying is that the $p$ value is a probability calcuated under the chi-squared $(\chi^2_1)$ distribution. You can think of a probability as being an area under a particular curve (think back to integrals in calculus).

The curve in question here is the chi-squared density enter image description here

The $p$ value is the blue shaded area under the curve I have plotted.

The curve is defined by: $f(x)=\dfrac{e^{-x/2}}{\sqrt{2x}\Gamma(1/2)}$

Now sure you can try to calculate this by hand to get p-value but most programming languages have built in functions to calculate are under density curves. In R you can do this:

> pchisq(1.75, df=1,lower.tail=FALSE)
[1] 0.1858767

edited May 18 '14 at 00:37

answered May 17 '14 at 20:28

bdeonovic

10,127

1

Sorry, are you sure that it is a two-sided p-value? – Sergio May 17 '14 at 21:13
The p value noted in OP's question looks like one-sided because it matches the one-sided calculation i did in R. – bdeonovic May 17 '14 at 21:24
I'd say that it is one-sided, because the chi-squared support is $[0,+\infty]$, so pchisq(-1.75,1)=0. Am I wrong? – Sergio May 17 '14 at 21:29
who said anything about -1.75? – bdeonovic May 17 '14 at 21:36
I mean: where is the other side? – Sergio May 17 '14 at 21:45
This section of the Wikipedia article about one- and two-sided tests expounds my doubt. – Sergio May 17 '14 at 21:57
I guess you can consider it a one-sided p value then if that makes you feel better. – bdeonovic May 17 '14 at 22:31
2

I'd fell better if $p=2\text{Pr}(\chi_1^2>1.75)=0.1859$ ;-) – Sergio May 17 '14 at 22:37
1

Regardless of semantics (is the test two-sided because it has power against departures from the null hypothesis in either of two directions, or one-sided because one tail area of the test statistic's density gives its size?) the expression $p = 2\text{Pr}\left{ \chi^2_1 > x_{obs}\right}$ is simply wrong - out by a factor of 2 as @Sergio says. – Scortchi - Reinstate Monica May 18 '14 at 00:29
I'm confused so am I not supposed to work out P? I'll post the full question. Is there no concrete formula for working out P in an exam? – jshbrntt May 18 '14 at 11:24
yeah you most likely won't be able to compute the p-value by hand. It requires doing quite a complex integral in most situations – bdeonovic May 18 '14 at 12:45

Where does p-value come from in McNemar's Test?

1 Answers1