3

I have a query about McNemar's test. As far as I understand, the technique assumes that cells are independent. It tends to be used with group studies whereby counts refer to individuals, for example a study an intervention to cease smoking (see below).

macnemar_smoking_example_1

I have seen this technique used in single case studies where counts refer to items that an individual is tested on both before and after an intervention (see below). This creates a dependency between the cells, i.e. data from all cells belong to the same individual. This could potentially lead to pseudo-replication, i.e. the treating of data points which are related as independent. Does this render the Macnemar test less well-suited to single case studies? Thanks.

macnemar_items_example

ttnphns
  • 57,480
  • 49
  • 284
  • 501
Nick Riches
  • 561
  • Yes, dependency is within cell, e.g. 10 pre-intervention smokers and 10 post-intervention smokers are the same people. But I am assuming that there should not be dependencies between cells. – Nick Riches Aug 07 '20 at 11:22
  • The test statistic only refers to one of the diagonals. Is that what you are referring to? – Nick Riches Aug 07 '20 at 11:40
  • Sorry, the issue was simply that I hadn't correctly understood a particular aspect of your question. – Glen_b Aug 08 '20 at 00:37

1 Answers1

4

Yes, you can use your lower matrix in McNemar's test to assess if the intervention (or anything that is associated with it, see below) changed the likelihood of person A to correctly respond to items. (There obvisouly is an association, comparing 20 to 5).

The reason why this works is, although you have a second layer of clustering (not only are the items identical pre- and post-intervention, but also the person who scores on the items is identical), what changes is not the statistical procedure but the interpretation.

While in the smoker/non-smoker case, you are using frequentist statistics to generalize from a sample of people to a population of people, you are in the second case generalizing from a sample of items scored by Person A to a population of items scored by Person A. So what you obviously cannot do is generalize to other persons in the second case, which is logical given that you do a case study and are sampling within the same person.

But there is more. The design you described is a singe-case experiment, for which there exists an own (small) branch of statistics-focussed literature and their own associated problems and difficulties, see e.g.:

  • Heyvaert, M., & Onghena, P. (2014). Randomization tests for single-case experiments: State of the art, state of the science, and state of the application. Journal of Contextual Behavioral Science, 3(1), 51-64.

The say: "A researcher conducting a nonrandomized SCE [single-case experiment] has to be very careful when attributing changes in the outcomes to changes in the treatment conditions, because it is possible that the observed response trend of the participant might have been there without any treatment manipulation. However, in a randomized SCE the random assignment of measurement times to baseline and treatment conditions provides control for sources of bias"

So whether you can infer a causal effect of the intervention for person A from your data totally depends on your design, no matter what McNemar's test says. While it is easy to randomize between persons, it is less easy to randomize within the same person. The way people usually do this in single-case experiments is via random ordering in the time dimension.

Further, single-case experiments are often better analyzed with non-parametric tests (like McNemar's), because many assumptions of parametric tests are typically violated in single-case data.

In conclusion, with your data structure, be aware of the difficulty of making causal statements for within-person comparisons (internal validity problem), and also of the difficulty of making generalizations to other persons (external validity problem).

stefgehrig
  • 1,149