Making sense of a reviewer's "30 observed counts minimum per cell" requirement on a contingency table

Question

I'm perplexed by the comment of a reviewer, who has taken issue with the sample size from a study I conducted. He says, relatively to the 2X2 contingency table below and a (significant) 2-sample test for equality of proportions:

9 and 19 are too low, all cells should have 30 observations minimum, no statistical test will fix that.

Here is the contingency table in question:

Gender	characteristic X	absence of characteristic X
Men	19	194
Women	9	251

The hypothesis here was that there is a difference of proportions between men and women relatively to the characteristic of interest (i.e. is the % of women with characteristic X different from the % of men with this characteristic?). If it's in any way relevant, the population studied here is of about 25,000 individuals.

A two-sided test of proportion (in R, prop.test(c(9,19), c(260, 213))) gives a p-value of 0.02105. (As a side note, a Fisher test gives a p-value = 0.01751.)

In my discipline and in the journal where I intend to publish, p-values < 0.05 are usually deemed as significant; the reviewer doesn't seem to take issue with this convention, so at first glance it is not a problem relative to significance thresholds and their relevance.

His point is not about model misspecification either, as the rationale for using only these two variables (gender/outcome) is given in the paper and he did not object to it.

His point seems to be that any given sample should follow this rule of "30 observed counts minimum by cell" in order to infer anything reliably.

I never read such a requirement before, and it is something I don't quite understand: in this case, even with a huge sample size, you may fail to satisfy this rule if an event is very rare in one of the groups you compare.

Now I still try to make sense of the reviewer's comment, in order to answer it or to understand a potential mistake I made here. It was my understanding that the sample size has no or limited impact on type-1 errors (see Can a small sample size cause type 1 error?). The reviewer insists, and makes me wonder if I'm somehow mistaken, if I misread something, if I conducted an incorrect test, or something else.

So, basically, I'm looking for a second opinion. Is there something incorrect in my analysis here? If not, do you know some reliable academic references that I can use to reinforce my answer to his comment? And if I'm mistaken, I'd be of course more than interested by an explanation.

Thanks,

score 4 · Accepted Answer · answered Feb 03 '23 at 17:58

4

The reviewer's "rule" of 30-per-cell is obviously outdated. I think their bigger point is that your samples are small (true) and you're likely under powered.

In R:

 pwr::pwr.2p2n.test(h = pwr::ES.h(p1 = (19/194),
                                  p2 = (9/251)),
                    n1 = (19+194),
                    n2 = (9+251),
                    sig.level = .05)
 difference of proportion power calculation for binomial distribution (arcsine transformation) 

          h = 0.2555792
         n1 = 213
         n2 = 260
  sig.level = 0.05
      power = 0.7897418
alternative = two.sided


NOTE: different sample sizes

You don't quite have even 80% power to detect this effect - that is, it just so happens to be the smallest effect you're powered to detect with this sample. Given that small samples are likely to over-estimate effect sizes, it's quite likely that the true effect in the population is smaller than this. And if that's the case, then you're under powered, and I'd be worried about this being a false-positive.

It can often be sufficient to appropriately caveat results throughout a manuscript.

answered Feb 03 '23 at 17:58

David B

1,532

Thanks, very clear explanation! – J-J-J Feb 03 '23 at 18:18
3

+1. "Outdated" is a kind charaterization, because it's difficult to imagine any date at which this was a reasonable rule of thumb. – whuber Feb 03 '23 at 18:22
1

I like the point about the power (though I see no evidence from their comment that power is what the reviewer had in mind). A common advice is to not calculate post-hoc power but here it's only used as a cautionary tale rather than as an important part of the analysis. Also, these samples are are only "small" because the probabilities are small. – dipetkov Feb 03 '23 at 21:00
2

In my experience, any of the "rules" people have about sample size come from incorrect assumptions about how power works. Perhaps I'm incorrect. I agree re post-hoc power, but note that I did not condone using said analysis to argue that they are powered to detect an effect. Instead I used it to highlight that their effect is right on the edge of what can detected with 80% power, which I think is extremely unlikely unless your effect estimate is inflated. – David B Feb 03 '23 at 21:01
1

Now it occurs to me that there might be more to this story. As the "minimum 30" is so much higher than even the most conservative standard recommendations, the reviewer actually might have a lot of experience with the trait being analyzed. So they know the trait is rare and even may have a good guess at how rare. The number 30 might very well come from that. – dipetkov Feb 03 '23 at 21:32
2

Maybe? 40 per cell for an ANOVA is a 10-year old rule-of-thumb in some psychology sub-fields, so 30 didn't strike me as particularly noteworthy. – David B Feb 03 '23 at 21:40
1

@dipetkov the reviewer did not elaborate on why this specific threshold (that's probably why I've been a bit confused by his comment). But anyway, regarding power, lesson learned for next time... About that, in case anyone is interested, I stumbled upon this related thread, with some good explanations on why low power is a danger not only relatively to Type II errors: https://stats.stackexchange.com/questions/176384/do-underpowered-studies-have-increased-likelihood-of-false-positives – J-J-J Feb 03 '23 at 21:48

Making sense of a reviewer's "30 observed counts minimum per cell" requirement on a contingency table

1 Answers1