4

Is there a way to calculate p-values from bootstrapped CIs extracted with the quantile function?

bs1 <- replicate(1000, sd(sample(c(10,23,21,12,14,14,13,14,15,25), 
                                 10, replace=TRUE)))
bs2 <- replicate(1000, sd(sample(c(10,11,10,13,13,13,14,19,12,23), 
                                 10, replace=TRUE)))

CIs <- quantile(bs1 - bs2, prob = c(0.025, 0.975))
AdamO
  • 62,637
locus
  • 1,593
  • https://stats.stackexchange.com/questions/315311/how-to-find-p-value-using-estimate-and-standard-error, maybe this can help? – Mathemagician777 Feb 15 '24 at 12:10
  • If the two vectors (10, 23, ...) and (10, 11, ...) are samples this seems to be a bootstrap problem and the following could be helpful https://stats.stackexchange.com/questions/20701/computing-p-value-using-bootstrap-with-r – AdamO Feb 15 '24 at 16:19

3 Answers3

12

To compute a p-value with a Monte Carlo test, you must sample values under the null hypothesis and compute the fraction of simulation results that fall beyond the value for the statistic computed from your sample. Apparently, your null hypothesis is that both populations have equal variance (or equal standard deviation). Your sampling, however, is not done under this assumption because your values bs1 stem from a population with variance 22.89 and your values for bs2 from a population with variance 15.36.

For correct ways to do this, see, e.g.,

Boos, Brownie: "Bootstrap Methods for Testing Homogeneity of Variances." Technometrics 31.1, pp. 69-82 (1989)

OTOH, if you have a confidence interval for the variance difference, what would a p-value be good for? Whenever a CI for the effect size can be computed, I would argue that p-values should not be reported, which leaves p-values for the few tests that do not have a meaningful corresponding effect size (like, e.g., goodness-of-fit tests).

cdalitz
  • 5,132
  • "Apparently, your null hypothesis is that both populations have equal variance (or equal standard deviation). Your sampling, however, is not done under this assumption because your values bs1 stem from a population with variance 22.89 and your values for bs2 from a population with variance 15.36." It seems like you're confusing population variance with sample variance. Under homoscedasticity, it's quite likely to observe variances at least this discrepant. – AdamO Feb 15 '24 at 14:58
  • @AdamO Yes, the sample variances will differ, even if the true variances are equal. But for computing a p-value, the sampling must be done from populations with equal variances, i.e. under the null. The given reference by Boos & Brownie achieves the null hypothesis by sampling from $x_1-\overline{x},\ldots,x_n-\overline{x},y_1-\overline{y},\ldots, y_n-\overline{y}$. – cdalitz Feb 15 '24 at 15:07
3

A CI can be constructed without a hypothesis. But a $p$-value requires you to state one. Consider, for one, that a difference in standard deviations is not a natural (interpretable) quantity but a ratio is, a ratio of normal sample variances for instance takes an F distribution.

In that case, we begin our line of inquiry to ask what the sampling distribution of the variance ratio is under the null hypothesis. As user Cdalitz points out, it is a common problem in bootstrap testing that the null hypothesis is not trivial to calculate. In the answer I link, a popular answer provides an alternative approach to calculating a $p$-value using resampling statistics if not bootstrap per se. Rather than resampling data, permute the ID labels to obtain such a distribution.

In your data:

df <- data.frame(
  i = rep(0:1, each=10),
  x = c(10,23,21,12,14,14,13,14,15,25,
  10,11,10,13,13,13,14,19,12,23)
)

re <- exp(diff(log(with(df, tapply(x, i, var)))))


pre <- replicate(10000, {
  df$i <- sample(df$i)
  exp(diff(log(with(df, tapply(x, i, var)))))
})
pre <- c(re, pre)
hist(pre)
abline(v=c(re, 1/re), col=c('blue', 'red'))


(mean(pre > re) + mean(pre < 1/re))/2

gives a permutation p-value of 0.72 which is equally as pessimistic as your CI for difference which spans from -2.2 to 3.4.

AdamO
  • 62,637
  • "A CI can be constructed without a hypothesis. But a p-value requires you to state one." Confidence intervals are very much related to p-values. Every method to compute a confidence interval defines a method to compute a p-value. https://stats.stackexchange.com/questions/561578 – Sextus Empiricus Feb 15 '24 at 20:59
  • @SextusEmpiricus you say this, but suppose I am developing a biosimilar drug and wish to claim it is similar to the approved mainline treatment, so I just present two CIs and see they largely overlap. Can I reasonably say my drug is at least as good as the approved treatment? Explicitly stating a hypothesis is a critical line of scientific inquiry, whereas a CI can sometimes just be a descriptive statistic. – AdamO Feb 15 '24 at 21:31
  • Confidence intervals can be loosely speaking used as a descriptive statistic, but they still need to be derived from the principle that defines confidence intervals, which always relate to p-values. Also, a p-value can be just as well a descriptive statistic, I am not sure what the point is about the idea that it can be a descriptive statistic. – Sextus Empiricus Feb 15 '24 at 21:33
  • "suppose I am developing a biosimilar drug and wish to claim it is similar to the approved mainline treatment" In such a case, the confidence intervals/distribution for the difference can be turned into p-values. – Sextus Empiricus Feb 15 '24 at 21:38
1

The estimated distribution from a bootstrap method does not always give sufficient information to compute a p-value, and neither a confidence interval.

  • One problem is that p-values (and confidence intervals) relate to hypothetical values of the distribution parameters. The shape of the distribution given those hypothetical values, can be different from what is observed in the sample.

    When estimating a distribution based on a bootstrap distribution, then often an assumption is made about the transformation between the bootstrap distribution and the distribution to be estimated. For example: a shift or scaling of the bootstrap distribution when the means of the bootstrap distribution and the mean of distribution to be estimated is different.

    A simple case of a discrepancy is the use of a bootstrap distribution to determine confidence intervals for the maximum $\theta$ of a uniform distribution $x \sim U(0,\theta)$. The observed sample will almost never contain the value $\theta$ and that value will be almost always outside the distribution estimated with the bootstrapping.

  • Another problem is that a bootstrap distribution is an estimate of the true distribution and may have some bias.

    For example the standard deviation of a bootstrap sample will be on average smaller than the true deviation, and requires a correction. This is similar to the use of a Bessel correction.

If you apply appropriate corrections and transformations then you can compute p-values based on the bootstrap distribution.

Sextus Empiricus
  • 77,915