0

I'm a little confused with the meaning of a statistically significant result. Does statistically significant just mean it's a phenomenon worth exploring given that there is a high likelihood of seeing a value as extreme as the observed value? And we should do further tests to understand the phenomenon?

Richard Hardy
  • 67,272
WheelyWonka
  • 3
  • 3
    Click on the statistical-significance tag to see its description. It tells what statistical significance is. – Richard Hardy Jun 23 '21 at 16:02
  • Thanks for the comment @RichardHardy. The tag describes the methodology to determine if a desired statistic is statistical significant. What I'm wondering is whether determining if something is statistically significant means something. – WheelyWonka Jun 23 '21 at 16:12
  • 2
    It means that the p-value we found is below an arbitrary value (often 0.05) – Robert Long Jun 23 '21 at 16:33
  • Thanks @DemetriPananos I've read your answer on the question. So, what I can take away from it is that statistically significant is just having the evidence to evaluate what I want to do with the null hypothesis and nothing more? – WheelyWonka Jun 23 '21 at 16:42

1 Answers1

2

In frequentist statistics we regularly do hypothesis tests that yield a p-value. The p-value tells us the probability of observing data at least as extreme as the data we actually observed, if in fact the null hypothesis is true. If the p-value is smaller than a pre-determined level, then we say that we found a "statistically significant" result.

The mistake that many people make is placing the desire to find a "significant" result above almost anything else, when in fact there are other very important considerations. One way I like to explain this is by thinking about a hypothetic trial for a new drug that is designed to lower blood pressure. Let's imagine that we find the drug reduces blood pressure by 0.2mm HG and the p-value is 0.049, with a pre-chosen significance level of 0.05. So we have found a statistically significant result. However, 0.2mm HG is a very small effect size that clinicians would find to be completely meaningless. So statistical significance should always be considered alongside practical or clinical significance.

Robert Long
  • 60,630
  • Thanks Robert for the answer. Correct me if I'm wrong, so a statistically significant result has no intrinsic meaning but only serves to tell us more about the null hypothesis? – WheelyWonka Jun 23 '21 at 16:50
  • @RenMizukiIsaDataScientist What is precisely is "intrinsic meaning in contrast to just "meaning"? – Galen Jun 23 '21 at 16:52
  • @Galen Sorry for making it more confusing. The question is: Other than telling us whether to reject/accept the null hypothesis, is there any other purpose? – WheelyWonka Jun 23 '21 at 16:57
  • From hypothesis testing more generally, you can obtain effect sizes that are sometimes meaningful. But the statistical significance part is just pertaining to the rejection of the null hypothesis. – Galen Jun 23 '21 at 17:01
  • Alright. Thanks @Galen! Appreciate the help – WheelyWonka Jun 23 '21 at 17:03