Power paradox: overestimated effect size in low-powered study, but the estimator is unbiased

Question

If we have an underpowered study but manage to reject the null hypothesis, anyway, it makes sense to wonder if we have overestimated the effect size.

However, such a concern seems unwarranted if we use an unbiased estimator of the effect size (such as $\bar X_1-\bar X_2$ to estimate $\mu_1-\mu_2$).

At the same time, there is an appeal to speculating that we must have lucked into a particularly large observed effect to have managed to reject despite the low power.

This seems like a paradox. Does it have a resolution?

Answer 1

It seems to me that an underpowered study by definition is unlikely to give a small p-value against the null. Consequently, if you do get a small p-value it is likely that you are overestimating the true effect size.

However, if you look at all estimates from repeated experiments, regardless of their significance threshold, you do get an unbiased overall estimate. This should reconcile your paradox? Here's a simulation to illustrate:

We repeat an underpowered experiments 10,000 times:

set.seed(1234)
es <- 0.1 # The true difference
n <- 5    # and a small sample size
est <- rep(NA, 10000)
p <- rep(NA, length(est))
for(i in 1:length(est)) {
    a <- rnorm(n, mean=0)
    b <- rnorm(n, mean=0 + es)
    tt <- t.test(a, b)
    est[i] <- diff(tt$estimate)
    p[i] <- tt$p.value
}

If you consider only experiments with p < 0.05 you get extreme estimates (blue line is the true value). Note that some estimates are extreme and are also in the wrong direction (those on the left of the blue line):

hist(est[p < 0.05], xlab='Estmates where p < 0.05', main='')
abline(v=es, col='blue', lty='dashed')

Nevertheless the estimatator is unbiased across the 10,000 experiments:

mean(est)
[1] 0.1002
Count of over- and under-estimating experiments:
length(est[est > es])
[1] 5056
length(est[est <= es])
[1] 4944

Answer 2

To resolve the issue of bias, note that, when we consider the effect size in a test that rejects, we no longer consider the entire distribution of $\hat\theta$ that estimates $\theta$ but $\hat\theta\vert\text{reject }H_0$, and there is no reason to expect this latter distribution to have the unbiasedness that $\hat\theta$ has.

Regarding the issue of being "underpowered", it is true that a formal definition of this term would be nice. Note, however, that as power increases, the estimation bias in estimates corresponding to rejected null hypotheses decreases.

library(pwr)
library(ggplot2)
set.seed(2022)
Ns <- seq(50, 2000, 50)
B <- 10000
powers <- biases <- ratio_biases <- rep(NA, length(Ns))
effect_size <- 0.1
for (i in 1:length(Ns)){
powers[i]<- pwr::pwr.t.test(
    n = Ns[i], 
    d = effect_size, 
    type = "one.sample"
    )$power
observed_sizes_conditional <- rep(NA, length(B))
  for (j in 1:B){
x &lt;- rnorm(Ns[i], effect_size, 1)

pval &lt;- t.test(x)$p.value

if (pval &lt;= 0.05){
  observed_sizes_conditional[j] &lt;- mean(x)
}

observed_sizes_conditional &lt;- observed_sizes_conditional[                  
  which(is.na(observed_sizes_conditional) ==  F)
  ]

ratio_biases[i] &lt;- mean(observed_sizes_conditional)/effect_size
biases[i] &lt;- mean(observed_sizes_conditional) - effect_size


}
print(paste(i, "of", length(Ns)))
}
d1 <- data.frame(
  Power = powers,
  Bias = biases,
  Statistic = "Standard Bias"
)
d2 <- data.frame(
  Power = powers,
  Bias = ratio_biases,
  Statistic = "Ratio Bias"
)
d <- rbind(d1, d2)
ggplot(d, aes(x = Power, y = Bias, col = Statistic)) +
  geom_line() +
  geom_point() +
  facet_grid(rows = vars(Statistic), scale = "free_y") +
  theme_bw() + theme(legend.position="none")

I do not know the correct term for what I mean by "ratio bias", but I mean $\dfrac{\mathbb E[\hat\theta]}{\theta}$. Since the effect size is not zero, this fraction is defined.

This makes sense for the t-test, where the standard error will be larger for a smaller sample size (less power), requiring a larger observed effect to reach significance.

By showing this, we avoid that irritating issue of defining what an "underpowered" study means and just show that more power means less estimation bias. This explains what is happening in the linked question, where a reviewer asked an author for the power of the test in order to screen for gross bias in the conditional estimator $\hat\theta\vert\text{reject }H_0$. If the power is low, the graphs above suggest that the bias will be high, but high power makes the bias nearly vanish, hence the reviewer wanting high power.

Answer 3

You may have an estimator $\hat\theta$ that is (unconditionally) unbiased for its target: $\mathbb{E}(\hat\theta)=\theta$. The absolute value of the estimator $|\hat\theta|$ may also be (unconditionally) unbiased for the absolute value of the target: $\mathbb{E}(|\hat\theta|)=|\theta|$. (The absolute value rather than the raw value is relevant when considering effect size.)

However, once you condition on statistical significance of the estimate, the absolute value of the conditional estimator will generally no longer be unbiased for the absolute value of the target: $\mathbb{E}(|\hat\theta|\mid \hat\theta\text{ is stat. signif. at }\alpha\text{ level})\neq|\theta|$.

_{(I had struggled with a similar question over here: Understanding Gelman & Carlin "Beyond Power Calculations: ..." (2014). The issue was not really the essence but rather presentation. In the beginning it was not immediately clear to me that Gelman & Carlin were actually conditioning on statistical significance.)}

Answer 4

Possibly the following image might shed some light

Given that the null hypothesis is true, there will always be an $\alpha\%$ chance to reject the null hypothesis, no matter what the power of a test is*.

But the power of the test makes the overall picture a lot more different. Possibly the paradox stems from gazing too much exclusively at the null hypothesis and p-value.

---

^{*Or actually the percentage to reject might be a bit higher because the hypothesis test is based on a theoretical model for the error and the reality might be different (sampling errors like outliers or correlation between measurements).}

Answer 5

It's not a paradox. You may call it a dilemma, or more precisely an unknown. You have correctly narrowed it down to the two possible outcomes: you are either really "lucky", or the assumptions behind the power calculation are incorrect. There is no way to know which is which based on the results of one study alone. These considerations matter even for well powered studies with statistically significant findings.

Answer 6

You have hit on the same question discussed in the well-known Why Most Published Research Findings Are False paper. If you do a lot of experiments as a scientific community and quite a few of the tested null hypotheses are true (i.e. people try to show a whole lot of effects that aren't really there, while some are), then "underpowered" studies are more likely to produce false positive findings than "well-powered" studies. Similarly, once one conditions on statistical significance, point estimates are biased away from where you put your null hypothesis. This bias is larger, the more underpowered a study is.

You might critique this by saying that null hypotheses are rarely exactly true, but the exact same things happen when you instead look at a set-up where many effects are very small and only a few are big.

People worry about this a lot in drug development, where large companies will run early stage proof of concept studies (you can look at those as a kind of screening tool for deciding which projects to pursue further) for many potentially promising new drugs (of which most will not have a meaningful effect on the disease of interest). It is important for these studies to not be completely underpowered, because otherwise "positive" proof of concept results will become useless as a tool for prioritizing which drugs to study further.