Is statistical insignificance fatal?

Question

I apologize if this question has been done to death, but as a non-statistician I really don't know what is the bottom line takeaway. I am looking at a sample of 30,000 individuals who were the subject of an economic intervention. This intervention produces a mean increase in the annual income of the subjects of $2000. However the result is not statistically significant at usual levels. From a statistical viewpoint, has the intervention failed, and if not what further can be done?

Answer 1

Statistical insignificance does not mean that the effect being tested for does not exist, but rather, that the data that was observed does not furnish strong evidence for the existence of that effect.

For example, if you have an unloaded six-sided die, but the numbers on its faces are {1,2,3,4,5,5} instead of {1,2,3,4,5,6}, and you roll it only 3 times, it may not be evident through such a small sample size that the die would give you more fives than ones. That doesn't mean the die isn't different than a normal die (after all, we have the benefit of inspecting it and we can clearly see it is different)--it may simply be that we need to collect more data about the die's observed behavior in order to make a statistically significant inference about the intrinsic properties of the die.

Analogously, it may be that even a sample size of 30000 may not be sufficient to detect a difference in the behavior of your population under two treatments, because your statistical test has low power. Or, maybe the truth is that the mean increase you're observing is actually due to random chance and no effect truly exists. Since you have not specified your tolerance for Type I error, I can't really speak to that.

The takeaway here is that failure to detect significance doesn't mean no effect exists--it simply means that, by random chance or by lack of power, the data furnishes insufficient evidence to claim that the hypothesized effect exists with a high degree of confidence.

Answer 2

Well, this is certainly not good news. Sorry.

Your results don not provide any evidence for the existence of an effect. The effect, of course, might still exist: it could be smaller or more variable than you expected, or your experiment was somehow flawed and failed to detect it.

So, what can you do now?

0) Check your data. Make sure nothing silly has happened. Missing values sometimes get coded as 0s/-1s/99s, and these numbers obviously shouldn't be entered into your analysis as actual values. Similarly, if you're randomizing people to treatments/controls, make sure these groups are actually similar. People get bitten by these sorts of bugs all the time.

1) Perform a power analysis. Ideally, you would have performed one before beginning the project, but doing one now can still help you determine whether your experiment, as performed, would have a reasonable chance of detecting your expected effect. If not, (perhaps your drop-out/noncompliance rate was very high), you might want to perform a larger experiment.

You should not add subjects, run the analysis, and repeat until your result becomes significant, but there are lots of strategies for mitigating the problems associated with taking multiple "looks" at your data.

2) Look at sub-groups and covariates. Perhaps your proposed intervention works best in a specific geographical region, or for younger families, or whatever. In general, it would best to specify all of these comparisons ahead of time, since exploiting "experimenter degrees of freedom" can dramatically increase the false positive rate.

That said, there's nothing wrong with looking per se. You just need to be upfront about the fact that these are post-hoc/exploratory analyses, and provide weaker evidence than an explicitly confirmatory study. Obviously, it helps a lot if you can identify plausible reasons for why the subgroups differ. If you find a hugely significant effect in the North, but nothing in the drought-stricken, war-ravaged South, then you are in pretty good shape. On the other hand, I'd be a lot more skeptical about a claim that it works on subgroups of people born during full moons but only at high tide :-)

If you do find something, you may be tempted to publish right away. Many people do, but your argument would be much stronger if you could confirm it in a second sample. As a compromise, consider holding out some of your data as a validation set; use some of the data to look for covariates and the validation set to confirm your final model.

3) Could a null result be informative? If previous work has found similar effects, it may be useful to see if you identify factors that explain why they weren't repeated in your population. Publishing null results/failures-to-replicate is often tricky because one needs to convince reviewers that your experiment is sufficiently well-designed and well-powered to detect the sought-after effect. With $n=30,000$ however, you are probably in pretty good shape on that front.

Good luck!

Answer 3

Regarding the title question: Categorically, no. In your case, not enough info, hence my comment and downvote. Also, IMO, questions that conflate statistical and practical significance have been done half-to-death here, and you haven't said enough to make your question unique. Please edit; I'll undo my downvote if I see improvement (it's locked now), and probably upvote if it's substantial. Your question addresses a common, important misconception that deserves being done the rest of the way to death, but as is, it's hard to say anything new about your situation that would make it a useful example.

From a statistical viewpoint, has the intervention failed, and if not what further can be done?

Again, what have you done so far? It's also quite possible that your analysis has failed, to borrow your term (IMO, "failed" is clearly too harsh in both cases). This is why I asked about your test. There's a fair amount of controversy surrounding pre-post analysis options, and random sampling or lack thereof is relevant to the choice of analytic options (see "Best practice when analysing pre-post treatment-control designs"). This is why I asked about a control group.

If your choice of test can be improved, do that (obviously). In addition to checking your data (as @MattKrause wisely suggested), check your test's assumptions. There are quite a few involved in the usual pre-post designs, and they're violated often.

• Normal distributions are likely to be poor models, especially for change scores and financial data. Consider nonparametric analyses.
• Heteroskedasticity is common, especially without random selection or with a partially stochastic intervention. Some tests are more sensitive to this than others – especially the conventional ones.
• Conventional ANCOVA assumes no interaction between interventions and covariates. If baseline income affects the viability of the intervention, you should probably use moderated regression instead $(\text{Final Income = Baseline Income + Intervention? + Interaction + Error}$, basically), assuming you do have a control group. If you don't, do you have more than 2 times?

What other info about your individuals do you have? Exploring covariates and moderators is a good way to reduce the amount of statistical "noise" (error) your intervention's "signal" (effect) has to overwhelm for your test to "detect" it (support rejection of the null). If you can explain a lot of variance by means other than your intervention, or explain why your intervention doesn't affect everyone equally, you might get a better sense of how big your intervention's effect really is, all else being equal – which is rarely the default state of nature. I believe this was the spirit of Matt's suggestion #2.

Regarding his caveat, don't be afraid to explore covariates and moderators you haven't specified in advance; just adopt an exploratory mindset and acknowledge this epistemological transition explicitly in any report you publish. The crucial point that bears repeating about statistical and practical significance is that their overlap is generally limited. Much of the practical significance of statistical significance is in what you intend to make of it. If you're seeking evidence to support further research (e.g., for a research grant), rejection of exploratory hypotheses may be enough. AFAIK, this is the only kind of practical significance that statistical significance is supposed to imply by default, and explains the choice of terminology historically: significant enough to justify more research.

If you're looking for a statistical viewpoint on whether your intervention is worthwhile, you're probably asking in the wrong way. Statistical significance is not intended to answer this by itself; it only directly represents an answer to a very specific question about a null hypothesis. I suppose this amounts to another suggestion: check your null hypothesis. It usually defaults to stating that the effect observed in your sample is due entirely to sampling error (i.e., effect of intervention = 0). Are you really interested in any change whatsoever? How consistent do you need it to be to justify the intervention? These questions partly decide the appropriate null; you need to answer them.

In confirmatory testing, you need to answer in advance. Since you've already run a test, any new tests of the same kind with different null hypotheses but the same sample would be exploratory. Unless you can collect another sample, it would probably be best to regard other kinds of tests as exploratory too. The strict sense of confirmatory hypothesis testing is particularly strict about the "no peeking" rule; IMO, this is a weakness of the hypothesis-testing paradigm as a whole. AFAIK, Bayesian analysis can be a little less strict about this, and might benefit you particularly if you can collect more data, because your current result could help inform your prior probability distribution.

Another way to approach the issue is by focusing on effect size and your confidence interval. $2K is a change in the direction you wanted, right? If your test's results meant what I think you think they meant, then there's a better than 5% chance you'd find a negative change if you were to repeat the study, assuming the intervention had no effect. If your investment had any positive effect at all, the probability is lower than your p value. If you're invested heavily enough in the prospect of the treatment, maybe you should replicate the study. Again, you know better than I what else affects that decision.

---

_{P.S. Despite my intro, I've managed to say plenty about this "half-dead" topic. Hopefully I've provided a useful summary of ideas other than those in preexisting answers, but I wouldn't be surprised if much of it isn't very useful to you personally. A big reason I wanted more info is that answering a vague question well practically necessitates covering a lot of unnecessary bases, which is kind of a waste of time. Nonetheless, if you grace us with an edit, I'll probably subsection off whatever no longer applies, and I might expand on what still does. It's evident from the incoming views that the question resonates with the audience here, so this could become a very useful question with a little more work.}

Answer 4

As a Bayesian, I often find myself interpreting experiments as positive evidence for the null hypothesis. I would ask the following things:

1. It's a mean difference of \$2,000, but what is that in terms of a standardized mean difference?
2. How big of a (standardized) mean difference would you have expected to observe if this intervention worked?
3. How precise is your estimate? If the estimate is +\$2000 +/- \$20,000, then you have not learned much -- perhaps there's too much variability to know if your intervention worked.
4. Now that you have observed this seemingly null effect in a pretty healthy sample of 30,000, might it be time to argue that you know place less probability in the intervention being effective?

Many considerations apply, of course. If you are looking at p = .02 when your traditional cutoff is .01, it would be foolish to conclude that the null hypothesis is true, as the data are probably fairly equiprobable under the two hypotheses.

Thus, I would suggest looking at Zoltan Dienes' webpage and his Bayes Factor calculator. By specifying your parameter estimate, its precision, and a plausible range of parameter values if your intervention worked, you could get a Bayes Factor telling you whether this is evidence that your intervention works or doesn't work, or whether there is no evidence one way or the other.

Of course, the other commenters' replies are important, too: check your model, check your data, etc. to make sure the parameter estimate you have is appropriate.

Answer 5

Yes it's fatal for economic intervention. Whoever you demonstrate your results to, will look at the significance and declare that intervention didn't work.

This is provided that you tested for significance properly. For instance, the samples with or without intervention are similar in a reasonable way, or that the differences were controlled for etc. There are all kinds of biases to be introduced inadvertently in these experiments, so you have to account for them somehow.