4

This question was inspired by this discussion I read recently. After obtaining our results, the assumptions from the models we used should be checked, otherwise these results may be deceiving.

Unfortunately, this important step is often skipped in practice in my field (health sciences), as I see it mostly because
i) checking model assumptions can be very time-consuming,
ii) it can make interpreting the results more difficult for the layman,
iii) it makes the results seem less attractive (e.g., if p-values are reduced after updating the statistical methodology in response to the violation of a model assumption, and
iv) analysts don’t know the assumptions, how to check them or what to do when they’re violated.

Whilst the consequences of the violation of these assumptions are not trivial, my question is how much would the literature actually benefit from every analyst verifying every assumption? On the one hand, this would very probably improve the overall quality of the literature, such as reducing false positive and false negative findings and the occurrence of inflated coefficients and deceptively small standard errors and p-values.

On the other hand, the slower time taken to perform rigorous statistical analysis would greatly lower the publication rate. If we assume that the false findings described above are non-systematic due to the use of different data sets and methodologies, then I can foresee how it could be beneficial to have a higher publication rate – even if the quality of the results is lower.

Is there any truth to this?

JED HK
  • 409
  • 6
    Having written a comment paper on a paper that performed tests of assumptions and ignored the results, it is potentially wasting the time of other researchers not to test your assumptions before publishing (on the bright side I had always wanted to publish a paper about dinosaurs ;o). Always be your own "reviewer 3" - find and correct your own problems before the reviewers get a chance to point them out. Better to write a good paper slowly than a bad one quickly. – Dikran Marsupial Mar 25 '24 at 14:07
  • 4
    "as the “true” results will eventually reveal themselves as the number of publications increases." this is incorrect, it isn't difficult to find very well cited papers (over many years) where the findings are incorrect or the method useless. This may be a better question for the Academia SE. – Dikran Marsupial Mar 25 '24 at 14:08
  • 1
    If you don't want to check assumptions, then use robust methods that use fewer assumptions – kjetil b halvorsen Mar 25 '24 at 14:47
  • Thanks for the discussion and thanks for both of your helpful answers on questions on this site :) – JED HK Mar 25 '24 at 16:58
  • @DikranMarsupial Edited, thanks! – JED HK Mar 25 '24 at 17:00
  • 2
    I think it would help to ask a more specific question. Which assumptions? Which test of those assumptions. For example, normality tests are rarely useful even though gaussian distributions are assumed. – Harvey Motulsky Mar 25 '24 at 17:20
  • @FrankHarrell has some recent comments on this topic. https://www.fharrell.com/post/assume/ – user78229 Mar 26 '24 at 01:00
  • 1
    @MikeHunter thanks for sharing! I love Harrell's work – JED HK Mar 26 '24 at 10:14

1 Answers1

13

Well, there is some truth to it, but it's problematic.

  1. It would lower the publication rate. OK. But getting rid of bad publications is a good thing. Too much garbage is published.

  2. Checking model assumptions is time consuming True, but it is time well spent.

  3. It can make interpreting results more difficult for the layman So, data analysts will learn ways to communicate.

  4. It can make results less attractive Ummm, if you rig your scale to say you weigh 10 pounds less than you do, you still need the same size pants.

  5. Analysts don't know .... Then train them or fire them. Better yet, hire analysts who know what they are doing.

  6. True results will appear as studies are replicated Not necessarily, or even usually. And, if they are, then you look like an idiot. Do you want to be the one who has the most retracted articles? Worse yet, someone might make an important decision based on your analysis. And the Challenger exploded, killing the crew.

Peter Flom
  • 119,535
  • 36
  • 175
  • 383