McNemar test with multiple scores for the same subject

Question

I am having a statistical issue:

• subjects receive a test with machine A and machine B.
• 3 experts classify whether the results of machine A and machine B are acceptable (1) or not (0).

From my understanding, the data are repeated measurements (as each subject gets tested by both machine A and B), so I should use McNemar test. However, McNemar test only applies to 2x2 matrices as far as I know.

So how do I handle this:

• I cluster the opinions of all 3 experts and create a 2x2 table (ICC between their measurements is >0.80, so you could argue their agreement is strong enough to do that) (data_all below).

OR

• I create 3 separate 2x2 McNemar tables (one for each expert) (data_1, data_2, data_3).

Example code:

data_all <- matrix(c(100, 21, 100, 10), ncol=2, byrow=T)
mcnemar.test(data_all)
data_1 <- matrix(c(33, 7, 34, 3), ncol=2, byrow=T)
mcnemar.test(data_1)
data_2 <- matrix(c(33, 7, 33, 4), ncol=2, byrow=T)
mcnemar.test(data_2)
data_3 <- matrix(c(34, 7, 33, 3), ncol=2, byrow=T)
mcnemar.test(data_3)

I am sorry if this question seems very basic.

Answer 1

You want Cochran's Q test. Just as the one way repeated measures ANOVA generalizes the paired t test to two or more measurements in the same individual (or more than two individuals matched per block), Cochran's Q test generalizes McNemars' test to two or more measurements in the same individual (or more than two individuals matched per block). Applying Cochran's Q test to a 2x2 design gives identical results to McNemar's test, and one may use either McNemar's test, or all the 2x2 Cochran's Q tests for post hoc pairwise comparisons following a Cochran's Q omnibus test for more than 2 groups.

Cochran's Q is implemented in software in:

• within R in the RVAideMemoire package as the as the cochran.qtest function,
• within SAS in proc FREQ,
• within SPSS in tests of k related samples, and
• within Stata as the cochranq package (within Stata type net describe cochranq, from(http://alexisdinno.com/stata)). This package implements effect size measures by Berry, et al, and by Serlin, et al.

References
Berry, K. J., Johnston, J. E., and Paul W. Mielke, J. (2007). An alternative measure of effect size for Cochran’s $Q$ test for related proportions. Perceptual and Motor Skills, 104:1236–1242.

Cochran, W. G. (1950). The comparison of percentages. Biometrika, 37(3/4):256–266.

Serlin, R. C., Carr, J., and Marascuillo, L. A. (2007). A measure of association for selected nonparametric procedures. Psychological Bulletin, 92:786–790.