2

Thanks for any answers in advance.

This question presumes a study has detected a difference between 2 groups, and has determined this difference to be statistically significant.

I'm reviewing a paper which wanted to recruit X patients based on a predicted difference between 2 groups of 5%. This was based on 85% power. They recruited X+300 patients so look to be adequaltely powered to detecta difference between the 2 groups.

Rather than finding a 5% difference as predicted, they actually find a 3% difference (which they still say is statistically significant). If they had started with a predicted 3% difference between the groups, 85% power would have been achieved at X+700 patients (arbitrary number to demonstrate the point). Since they only recruited X+300 patients, they would be 400 patients short of this new recruitment target and thus would be considered to be underpowered.

Does this matter? They have detected a statistially significant difference anyway, despite being underpowered. Or is this an issue of the result potentially being random chance and they are underpowered to be confident the difference is real?

user356816
  • 543

2 Answers2

6

For a 'significant' result to have 'low' power it has to be not very significant. There is a one-to-one relationship between the p-value of significance test and the power calculated from the sample size, observed effect size, and observed variance or standard deviation. (See this question for a lengthy discussion What is the post-hoc power in my experiment? How to calculate this? or go to the Hoenig & Heisey paper cited in the accepted answer.)

The scenario in the question probably involves calculation of power using the pre-data variability and the post data effect size. That is not usually helpful. An experimental result can be statistically 'significant' with a small observed effect size only if the observed SEM is small. That can happen when the variability of the sample is small or when the sample is large.

Michael Lew
  • 15,102
5

The study is underpowered assuming the observed effect is the true effect, which is probably not the case.

Anyway, the study seems to have done a very reasonable thing. They hypothesized an effect size, powered the study with adequate samples to detect that difference, and low and behold found something statistically significant. I don't see any wrong doing there.

That they estimated an effect smaller than their hypothesized effect size is also fine, this does not mean the study was underpowered. I would be more concerned if the study had very few samples and found an effect. The effect could be explained away by mismatch between model and assumptions and noisy estimates. Now, this does not exonerate the study of any wrong doing, it just doesn't sound like much to worry about from your description.

Demetri Pananos
  • 36,121