3

I understand the basics of Simpson's paradox, but I'm not yet confident that I'd always be able to "avoid" it, or to spot cases where others have failed to do so. To be more precise, I'm not sure I would always be able to spot situations in which some fallacious argument of the form

$$\forall i, P(A|C_i) > P(B|C_i) \Leftrightarrow P(A) > P(B),$$

or an equally fallacious variant thereof, is being invoked (most likely tacitly).

How often do such errors occur in the peer-reviewed literature? How often are authors called on them? Are there common but hard-to-detect forms of such arguments?

For example, I routinely come across "averages of averages" being treated as equivalent to "global averages". Is this a potential backdoor to Simpson's-paradox-like fallacies?

kjo
  • 1,967
  • 5
    Unless you have an oracle that can see when there's an important variable you haven't accounted for, I don't know that it's possible to have a foolproof way to know you avoided it - nor any reliable means of figuring out how often it happens. Well designed experiments, of course, are organized to avoid it (by randomization of treatment to subjects), but that's not always possible. The fallacious argument you wrote at the end there looks like it might also relate to some other aspects of the ecological fallacy as well. – Glen_b Nov 04 '14 at 04:41

1 Answers1

2

Since nobody have tried to answer so far, I give it a try. As Glen_b's comment tells you, you cannot expect a perfect answer, an algorithm, to find this. But I have found it useful in data analysis to try different things, like first doing univariate analysis, then using multivariable regression. Then look at the results. If some variables give strikingly different conclusions in those two analysis, then that is a sign to start thinking about what is happening, try to understand tyhe WHY, with your data, the two analysis gives different conclusions.

But such strategies can only be a start!

kjetil b halvorsen
  • 77,844