2

So I understand that saying that a p-value is the probability of a particular result coming about by chance is incorrect as per community consensus, the American Statistical Association, etc.

My internal understanding of the p-value is that it is the probability of getting a test statistic of a certain value or greater (depending on tailedness of test) in a hypothetical sampling distribution of said statistic where the distribution parameters are set to reflect the null hypothesis, which is often specified to be a non-relationship.

However, the problem I am facing is that to me, this does not conflict with the idea of "chance." To me, the spread of that hypothetical distribution is not necessarily just subject to the whims of the world, but could also be a function of observable variations in other variables. All I see that hypothetical distribution as representing is the possible spread of the test statistic, without making any claims about why the spread occurs in any particular way. So to me, "the chance of getting a value of X or greater" feels the same as "probability of getting a value of X or greater by chance over repeated random samples."

It occurred to me that the reason why I was having trouble seeing why saying "probability that result happened by chance" is categorically incorrect may be because I don't understand how people are using the word "chance." Or, alternatively, is there a different problem? Am I indeed understanding something about p-values incorrectly?

On a practical note, this is bleeding into my teaching, where I am struggling to find a way to explain the p-value to practice-oriented (public policy) introductory statistics students in a way that is reflective of what is actually happening re: the null hypothesis. I have seen discussions of this elsewhere (e.g., here), but I have not seen a discussion of the word "chance" in this context and what it means to people. I try to avoid saying "by chance," but I'm finding it difficult to find ways to explain it otherwise in a way that doesn't require me also teaching them about probability distributions, some ideas about areas under a curve, what "random" means in the statistical sense, etc.

RickyB
  • 1,145
  • 1
  • 12
  • 21
  • If you have a link to the ASA quote on this, please post it. It absolutely is by chance, GIVEN the underlying distribution mean. – Alex R. Nov 03 '17 at 18:16
  • 1
    The null hypothesis does not really mean no relationship, it can be any hypothesized baseline relationship, as long as it leads to a way to calculate the sampling distribution of your test statistic. The no relationship thing is just a human convention, but it is not necessary. – Matthew Drury Nov 03 '17 at 18:17
  • @AlexR. The statement was here: http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108#.Wfyy6GiPI2w ("P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.") – RickyB Nov 03 '17 at 18:19
  • @MatthewDrury. Oh agreed, that an important point. I've edited to reflect this acknowledgement. – RickyB Nov 03 '17 at 18:20
  • 2
    I see this phrase "probability the data were produced by chance alone" frequently used by non-statisticians posing as statistical experts. The problem with it is the failure to specify that the probability refers to the (almost certainly) counterfactual null hypothesis. Without clarity on this point, such statements leave it up to the reader to understand--usually incorrectly--exactly what state the world was presumed to be in when such a "chance" was computed. Moreover, many p-values are not probabilities of any particular states, but are suprema over a set of states. – whuber Nov 03 '17 at 18:33
  • @whuber. That makes sense. If that is the problem, is the solution then to tell people to stop teaching "by chance alone," or to make sure that people teach "by chance" in reference to a specific hypothesis? Is it just too risky to talk about chance because people will take it in the wrong direction? – RickyB Nov 03 '17 at 19:21
  • 3
    The problem is much more profound and difficult than that, unfortunately. One thing we learned at last month's Symposium on Statistical Inference is that no amount of teaching p-values, no matter how well done, is likely to prevent people from misunderstanding and abusing them. The proposals concerning what to do about this are all over the place, ranging from persuading journals to lower their threshold of significance to $0.005$; to using alternative methods based on maximum possible Bayes factors, confidence distributions, etc.; and everything else between. – whuber Nov 03 '17 at 19:25
  • 1
    @whuber. Thank you for this response. I have been struggling with this issue for some time now re: teaching and such, and I was concerned that I was missing something essential or obvious. Knowing how this conversation has been happening in the field is very useful. I think being more realistic about the extent to which I can totally prevent my students from abusing p-values is maybe a path forward, personally. – RickyB Nov 03 '17 at 19:32
  • @whuber Now that the semester is over, I am coming back to this. Do you know if there will be any future gatherings or discussions on this topic? I've finally gotten a chance to go through some of the material from the Symposium and would like to find out if there are ways to learn more in the future. – RickyB Jan 04 '18 at 01:30
  • There are plans to publish papers from the Symposium along with additional contributions by participants. (During the last couple of hours there were wide-ranging public discussions about many issues; some of the interlocutors plan to contribute papers to share their thoughts.) Look for that sometime this year. – whuber Jan 04 '18 at 14:01

1 Answers1

2

So I understand that saying that a p-value is the probability of a particular result coming about by chance is incorrect

I'm not sure it's incorrect. It depends on what you mean by something "coming about by chance", which isn't statistically precise terminology. People who have $p$-values described to them as "the probability that the result came about by chance" may think that, for example, if the $p$-value is low, the result is unlikely to have "come about by chance", and hence the estimate of the population effect must be very accurate. In reality, rejecting the null hypothesis doesn't imply your estimate is particularly accurate.

I try to avoid saying "by chance," but I'm finding it difficult to find ways to explain it otherwise in a way that doesn't require me also teaching them about probability distributions, some ideas about areas under a curve, what "random" means in the statistical sense, etc.

I don't think it's possible to explain a $p$-value in a way that's correct and not misleading without getting into the basics of statistics. You need to describe probability itself, distinguish between samples and populations, explain the notion of a parameter and a parametric family, etc.

Kodiologist
  • 20,116