14

I sometimes speak about statistical results to a popular audience, and the term "significant" can (understandably) be misunderstood. I sometimes want to say something like "the likelihood of seeing these results under the null hypothesis is small enough that there probably is something going on here, however the effect size is small enough that it's probably not worth worrying too much about."

Among technical audiences, I believe this could be summarized as "small but significant". Is there a short way I can communicate this to a non-technical audience?

mdewey
  • 17,806
Xodarap
  • 2,608
  • 18
    "Statistically detectable" is one of my favourites to get rid of the misleading term "significant", especially when talking to non-statisticians. – Michael M Jun 25 '20 at 17:03
  • 3
    @MichaelM I think you should expand on that and post as an answer. I like it a lot, too, and I am going to use it! – Dave Jun 25 '20 at 17:05
  • @MichaelM: I second Dave's proposal. – Stephan Kolassa Jun 25 '20 at 17:44
  • 3
    I also like "statistically detectable", especially for a lay audience. But you asked how to compactly communicate two things: that, plus that it's not important. So I'd suggest (words separated by slashes are meant to represent different options): "statistically detectable, but not biologically/physically/economically [insert your field here] meaningful/important/significant". – theorist Jun 26 '20 at 01:26
  • 2
    I think this question would be clearer if you wrote "significant but small," since what you are emphasizing is that the difference is negligible. – Max Jun 26 '20 at 04:18
  • Sure is: "small but statistically significant" – Glen_b Jun 26 '20 at 05:38
  • Agree with Max - when you say "small but significant" you are down-playing the small and emphasizing the significant, and when you say "significant but small" the emphasis is on the small. – Brian Stamper Jun 26 '20 at 16:12
  • 2
    @theorist: +1, I believe it captures the essence succinctly. Alternatively: "statistically detectable, but of no practical relevance". – Igor F. Jun 30 '20 at 07:27
  • @Max I think "significant but small" would confuse a lay audience, because when they hear "significant" they don't think "statistically significant", they think "important". So "significant but small" would end up communicating (to them) "important but small". To avoid this, you'd need "statistically significant but small", which I don't think is as good (for a lay audience) as something like "statistically detectable but unimportant". – theorist Jun 30 '20 at 07:48
  • @theorist I'm not talking about the lay audience; I'm saying that this question on SE is asking about the concept of a "significant but small" difference (a difference that is statistically significant but too small to worry about), not a "small but significant" difference (which reads to me as "small in absolute terms but worthy of statistical attention"). – Max Jun 30 '20 at 07:50
  • @Max Ah, I understand. But in that case I think the question would be clearer still if he phrased the title as "significant but unimportant," since what he wants to communicate to the audience is not that the difference is small per se, but more specifically that it's unimportant. That's why he doesn't say "small", but "small enough..." to not be important. There are results that are statistically significant, extraordinarily small, and yet extraordinarily important. The signals for the direct detection of gravity waves, in which the changes in mirror spacing are 1 part in 10^21, come to mind. – theorist Jun 30 '20 at 08:35

4 Answers4

14

If the audience knows what "statistical significant" and $p \le 0.05$ means, there is not much that can go wrong. But otherwise, I really like the fantastic suggestion by Jordan Ellenberg as an alternative to "statistically significant" in general:

[...] "statistically noticeable” or statistically detectable" instead of “statistically significant”! That would be truer to the meaning of the method [...] - Jordan Ellenberg in his book "How Not to Be Wrong: The Power of Mathematical Thinking"

Edit based on the short discussion in the comments: Note this answer does not specifically address the "small effect" situation but rather proposes an apt word for statistical significance in general where you don't have to fear that people take "statistically significant" as "relevant". In this way, you can distinguish in a clean and understandable way the two topics "hypothesis testing" and "effect size".

Michael M
  • 11,815
  • 5
  • 33
  • 50
  • -1, as this does not address the effect size issue in any way. With enough data, you might show that a new drug lowers cholesterol in a statistically significant manner by 0.000001%, which is a likely clinically meaningless finding. Simply stating that a difference is "statistically significant" gives no indication whatsoever whether the difference is meaningful or not. The headline "New Drug Statistically Significantly Lowers Cholesterol", while true, omits the more important part of the story, which is that the drug will have near-zero effect on human health. – Nuclear Hoagie Jun 26 '20 at 14:22
  • 3
    This is correct and originally a reason to post the term "detectable" only within a comment. Still, instead of using a misleading term to stress significance and then relativise its meaning for small effects, a strategy is to use an apt word from the beginning and then maybe throw in the fact that the effect is large/small (which is an aspect not part of the concept of testing). – Michael M Jun 26 '20 at 15:00
11

I have found the distinction between statistically significant and physically significant is often useful when communicating results to non-statisticians. The phrases clinically significant or practically significant may be preferred in some fields (thanks @Jelsema). You are describing a situation where the effect may be statistically significant, but physically insignificant.

As an aside, there is actually a push right now to stop (or limit) using the phrase "statistically significant" altogether. Some relevant reading:

Moving to a World Beyond "p < 0.05"

The Difference Between "Significant" and "Not Significant" is Not Itself Statistically Significant

Scientists rise up against statistical significance

References

Wasserstein, Ronald L., Allen L. Schirm, and Nicole A. Lazar. "Moving to a world beyond “p< 0.05”." (2019): 1-19.

Gelman, Andrew, and Hal Stern. "The difference between “significant” and “not significant” is not itself statistically significant." The American Statistician 60.4 (2006): 328-331.

Amrhein, Valentin, Sander Greenland, and Blake McShane. "Scientists rise up against statistical significance." (2019): 305-307.

knrumsey
  • 7,722
  • 4
    Just to tack on to the language aspect, with medical researchers I've used "clinically" significant to highlight the different between statistical results and practical meaning. – Jelsema Jun 25 '20 at 17:27
  • I much prefer "practically" (as opposed to "physically") here, to imply it's significant in practice (as opposed to in theory). It's also synonymous with "basically", which would also work if it's insignificant, but less so if it's significant. – Bernhard Barker Jun 26 '20 at 10:47
  • 1
    @Bernhard, thanks for the perspective! I often use "physically" because I work with a lot of engineers, but I understand that it may not be the best word-choice for most disciplines. – knrumsey Jun 26 '20 at 16:04
  • 2
    "practically significant" works well enough across many fields. But the bigger idea is to use the adverb that registers with the readership or audience. See also "economically", as in "the error is millions of dollars, but that is not economically significant". – Nick Cox Jun 30 '20 at 07:16
4

I would use something like, "We notice a difference that is extremely likely to be there and not a fluke. However, the difference has no practical meaning. They are effectively the same, like the distance to the sun versus the distance to the sun plus an inch."

The acknowledges the statistical significance ("not a fluke") while acknowledging the lack of practical significance ("no practical meaning").

Dave
  • 62,186
1

Referring to the tale of "The Princess and the Pea", you could say :

"There seems to be a pea under all these mattresses."

See also: https://en.wikipedia.org/wiki/The_Princess_and_the_Pea

dstrube
  • 11
  • 2
    But the pea mattered to the princess... – Dave Jun 27 '20 at 19:05
  • There are princesses who can detect peas under different beds in parallel universes, – Nick Cox Jun 30 '20 at 07:17
  • @Dave: Yes, and the statistician is here the princess, in the sense that the statistician was able to detect it. To all other folks in the world, the pea doesn't matter, because they cannot detect it. – Igor F. Jun 30 '20 at 07:18