2

I have a simple query regarding the construction of a confidence interval for a p-value. How do I construct such a confidence interval? Like we construct a C.I. for t-value ($\bar{X} \pm t \times(s/ \sqrt{n})$), can such kind of intervals be created for p-value?

I think (not completely sure) such C.I. for p-value is called replication interval. I tried googling but couldn't find anything relevant.

Dark_Knight
  • 569
  • Possibly related: http://stats.stackexchange.com/questions/250269 – amoeba Dec 09 '16 at 08:22
  • Also related: http://stats.stackexchange.com/questions/258143/estimating-population-p-value-pi-using-an-observed-p-value?noredirect=1&lq=1 – Tavrock Mar 17 '17 at 16:30
  • This question needs more background and details. It implies a setting in which (a) there is a true but unknown p-value and (b) somehow you are estimating that p-value with uncertainty. – whuber Oct 31 '23 at 21:02
  • @whuber. Is there ever a "true but unknown p-value"? I don't think so. A p-value is an assessment of a particular data set and null H. I can't think of a situation where there is a true (population) value. – Harvey Motulsky Oct 31 '23 at 21:36
  • This question was asked before (so is a duplicate) of this one (with good answers): https://stats.stackexchange.com/questions/254595/what-is-the-confidence-interval-of-a-p-value – Harvey Motulsky Oct 31 '23 at 22:34
  • @Harvey Yes! Bootstrapping is the standard example. There will be a specific p-value for the bootstrap statistic, but except for the smallest datasets it's never computed: it is only estimated through resampling. – whuber Nov 01 '23 at 12:54

2 Answers2

8

One constructs a confidence interval for a parameter of interest whose true value is unknown (the parameter is related to properties of the population). 95% of 95% confidence intervals contain the true value for the parameter.

A p-value, on the other hand, is an outcome of a random variable (its value depends on the sample). It makes no sense to speak of a confidence interval for a p-value, since it does not represent a parameter of unknown value. There is no such thing as the true p-value for a population.

Perhaps your question is asking about the distribution of p-values. For the non-discrete case, the p-value distribution is uniform [0-1], if the null hypothesis is correct.

Dean
  • 199
  • 1
    +1. Since OP mentioned "replication interval", this question (so far unanswered) might be relevant: http://stats.stackexchange.com/questions/250269 – amoeba Dec 09 '16 at 08:23
  • @amoeba Lots of helpful answers now about the distribution of possible p-values (different than confidence intervals) in that link. – Harvey Motulsky Nov 02 '23 at 16:29
0

Calculating confidence interval for p-value is a valid approach when you use the Monte Carlo simulation to obtain p-value (e.g., in R when you choose simulate.p.value=TRUE for chi-square test). In this case, p-value varies, so you may need to use, e.g., 99% C.I. To find out more please see C.R. Mehta and N.R. Patel, Exact tests, SPSS inc., 2012. How to calculate this in R, you may see an example in the following paper: https://rdcu.be/dpz18.

user17811129
  • 11
  • 1
    I don't think "confidence interval" is the right term to use for your approach. I believe you are making assumptions about variation, setting a sample size, and simulating many data sets. Then you can present a range of possible p-values. That can be useful, but is quite distinct from a confidence interval (as @dean wrote) that is trying to predict the uncertainty with which you know a population paramter. – Harvey Motulsky Oct 31 '23 at 15:06
  • @Harvey I believe this answer might reflect a different interpretation of the question than yours. It looks like a correct and helpful response subject to the conditions laid out in the first sentence. – whuber Oct 31 '23 at 21:03
  • @whuber It is all a matter of terminology. Calculating that interval is definitely helpful. But it seems more like a prediction interval rather than a confidence interval. Picky maybe, but using terminology correctly is needed for clear communication. – Harvey Motulsky Oct 31 '23 at 21:35