5

For the purpose of this post, accept that there is some critical threshold below which a finding is declared to be statistically significant.

The p-value is a statistic because it is a function of the data.

Statistics are random variables. We can put confidence intervals around statistics.

Why don't we put confidence intervals around p-values? Or, equivalently, why don't we test the hypothesis that the observed p-value is below 0.05, say?

One answer could be that it would be hard to calculate analytically, but bootstrapping could solve that issue.

amoeba
  • 104,745
Charlie
  • 14,062
  • 5
  • 44
  • 72
  • 5
    p-value is a characteristic of a given sample, not an estimate of something in the population to be able to bear some error. So p-value is an exact number, not an interval. p-value is interval only in Monte Carlo testing - because in such a testing not full information from the sample is used to compute p. – ttnphns Mar 08 '13 at 05:56
  • 2
    The p-value depends on the null hypothesis in addition to the data, whereas the data do not depend on the null hypothesis. – Alexis Jun 19 '15 at 17:15
  • 1
    Very relevant thread: https://stats.stackexchange.com/questions/254595 - see discussions in the comments. – amoeba Aug 18 '17 at 16:30

1 Answers1

2

Among other things, you might want to read: The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant

Wayne
  • 21,174
  • This is a bare link and does not describe the bare connection to the question. Though it is good reading. – Erik Mar 08 '13 at 08:34
  • I generally agree that one-sentence answers are bad, but that's all I have. I can't explain it myself, but know of a paper by a famous statistician who seems to address the point. – Wayne Mar 08 '13 at 12:47
  • @Wayne, I'm aware of the paper, thanks. But Gelman and Stern are arguing that estimates of a mean, say, between two samples may not be statistically different even though one of the estimates is significant while the other is not. I'm looking at a different issue: Why no confidence intervals around a p-value for a particular sample? – Charlie Mar 08 '13 at 23:37