Comparing Proportion of Positive Tests

Question

A condition or disease (D) is measured using two different methods (A and B) in a sample of 1,000 individuals from a population. Using method A, the percentage of positive cases is 25%, whereas method B yields 30%.

Is there a test of statistical significance I can apply to compare method A and method B? How can we test for whether these different percentages are wide enough to conclude that we are dealing with two methods of testing for D that are essentially different beyond statistical noise?

I was assuming that it would amount to testing two different proportions, but when I plug the data and analyze it with R I realize that the categorical options for method A and B are identical (i.e. positive and negative), and the comparison of proportions doesn't make sense.

It's a basic question, but I'm stuck... Here is the data:

  str(dat)
 'data.frame':  1000 obs. of  2 variables:
      $ Method.A: Factor w/ 2 levels "neg","pos": 1 1 1 1 2 2 1 1 1 1 ...
  $ Method.B: Factor w/ 2 levels "neg","pos": 1 1 1 1 2 2 1 1 1 1 ...


    summary(dat)
    Method.A  Method.B 
    neg:574   neg:488  
    pos:426   pos:512 


   countdf
      Method.A Method.B Freq
    1      neg      neg  302
    2      pos      neg  186
    3      neg      pos  272
    4      pos      pos  240

Clearly the methods are different, with Method B tending to show more positives (True or False), but if I try to run a Chi-square test it will show a statistically significant p value, implying correlation between both Methods, which of course it's there because both methods attempt to detect the disease, and neither one is awful.

What I need is to show that they are different in the way they go about detecting D, because they can't possibly be equivalent when the positives for Method B are 512 while Method A only flags 426 subjects as positive.

Just for completeness:

Percentage table:
        Method.B
Method.A  neg  pos Total Count
     neg 52.6 47.4   100   574
     pos 43.7 56.3   100   426

    2-sample test for equality of proportions without continuity
    correction

data:  .Table
X-squared = 7.8415, df = 1, p-value = 0.005106
alternative hypothesis: two.sided
95 percent confidence interval:
 0.02716934 0.15185603
sample estimates:
   prop 1    prop 2 
0.5261324 0.4366197

---

Following the lead of another thread of discussion vis-a-vis Agresti's matched observations (Testing paired frequencies for independence), I went one step further in my answer, and came up with the following results.

Of note, I realize that I am treating these values as ordinal, and methodologically it is a stretch, but I'm getting the same result applying a poisson glm.

two_by_two<-matrix(c(240,186,272,302),nrow=2,byrow=T)
dimnames(two_by_two) <- list(c("pos","neg"),c("pos","neg"))
names(dimnames(two_by_two)) <-c("Method.A","Method.B")
require(vcd)
library(gnm)
tab <- data.frame(counts=c(two_by_two), Method.A=gl(2,1,4,labels=c("neg","pos")),
                     Method.B=gl(2,2,4,labels=c("neg","pos")))
tab$scores <- rep(c("neg","pos"), each=2)
summary(fit <- gnm(counts ~ scores + Symm(Method.A,Method.B), data=tab, 
                       family=poisson))

Call:

gnm(formula = counts ~ scores + Symm(Method.A, Method.B), family = poisson, 
    data = tab)

Deviance Residuals: 
[1]  0  0  0  0

Coefficients:
                               Estimate Std. Error z value Pr(>|z|)    
(Intercept)                     5.48064    0.06455  84.906  < 2e-16 ***
scorespos                      -0.38006    0.09515  -3.994 6.48e-05 ***
Symm(Method.A, Method.B)negpos  0.12516    0.08856   1.413    0.158    
Symm(Method.A, Method.B)pospos  0.60984    0.12857   4.743 2.10e-06 *** 

And my follow up question is whether it would be appropriate to interpret the results as: The probability that Method.B is “neg” when Method.A scores “pos” is exp(-0.38006) = 0.68382 times the probability that Method.B has a “pos” score while Method.A has a “neg” score?

Here are for comparison two poisson glm fits:

summary(fit.symm <- glm(counts ~ scores + Symm(Method.A,Method.B), 
                        family = poisson(log),data = tab))

Coefficients:
                               Estimate Std. Error z value Pr(>|z|)    
(Intercept)                     5.48064    0.06455  84.906  < 2e-16 ***
scorespos                      -0.38006    0.09515  -3.994 6.48e-05 ***
Symm(Method.A, Method.B)negpos  0.12516    0.08856   1.413    0.158    
Symm(Method.A, Method.B)pospos  0.60984    0.12857   4.743 2.10e-06 ***

And,

    summary(fit.symm <- glm(counts ~ Symm(Method.A,Method.B), 
                            family = poisson(log),data = tab))
Call:
glm(formula = counts ~ Symm(Method.A, Method.B), family = poisson(log), 
    data = tab)

Deviance Residuals: 
     1       2       3       4  
 0.000   2.759  -2.938   0.000  

Coefficients:
                               Estimate Std. Error z value Pr(>|z|)    
(Intercept)                     5.48064    0.06455  84.906  < 2e-16 ***
Symm(Method.A, Method.B)negpos -0.04692    0.07969  -0.589  0.55602    
Symm(Method.A, Method.B)pospos  0.22979    0.08647   2.657  0.00788 ** 

Is this last model best? Would then the conclusion be: The probability that Method.B is “neg” when Method.A scores “pos” is exp(-0.04692) = 0.954 times the probability that Method.B has a “pos” score while Method.A has a “neg” score?

Answer 1

OK. Here's what I got so far, and I hope this is the answer:

The right test for this situation in which the same data is looked at with different prisms, or there is a before and after treatment setup, and with the variables being categorical leading to proportions and 2 x 2 contingency tables, is the McNemar test.

In this test what we look to see if whether the effect of different "methods" (to follow the terminology in my question above), or the changes between pre- and post-treatment counts affects the diagonal columns of the contingency table:

        Method.B
Method.A neg pos
     neg 302 272
     pos 186 240

If both methods were interchangeable, one would like to see a table such as:

        Method.B
Method.A neg pos
     neg a    0
     pos 0    d

When one test yields a negative result, so does the other test (or "method"); same for positive results.

So it is the magnitude of these diagonal cells when they are different from zero (let's call them b and c) that are the fundamental of the McNemar test:

The McNemar test statistic is:

$\chi^2 = {(b-c)^2 \over b+c}$.

So the more discrepant these diagonal values are $(b - c)^2$, the larger the $\chi^2$ value, and the more likely will it be that we are dealing with different "methods" - the proportions of Method A are different from Method B.

To call the test in R:

two_by_two <- table(dat)

two_by_two
        Method.B
Method.A neg pos
     neg 302 272
     pos 186 240

 mcnemar.test(two_by_two, correct = F) #No correction generally needed.

McNemar's Chi-squared test

data:  two_by_two
McNemar's chi-squared = 16.1485, df = 1, p-value = 5.857e-05

And there it is: Method B tends to yield more positive results when measuring the presence of disease, D, and the p value of accepting $H_1$ is ridiculously small.