3

I am working on a project which uses a publication that appeared in  print in 1998 and reported a completed double blind randomized clinical trial. 

Due to the confidential nature of the project I can't give too much detail. The protocol for the clinical trial planned to have three co-primary endpoints. A multiple comparisons procedure was not described in the protocol. In the publication Each of the co-primary endpoints was statistically significant at the 0.05 level. 

I'd appreciate thoughts as to whether or not a "multiple comparisons procedure" (MCP) can be "retrospectively" applied to that statistical analysis in this example to a publication completed over a decade in the . 

In advance, my view on MCP's is that they should be prospectively planned in order to control Type I error and that there is no justification for retrospective application of an MCP. 

thanks in advance

Chris Barker
  • 31
  • The trial is a randomized clinical trial. Certain tests were planned in advance and statistical analyses done (several p-values quoted. I would agree that it would not be right to do new analyses that weren't planned like subgroup analysis that historically are controversial and have been included in medical research journal publications including major RCTs. But what about adjusting the p-values for multiple tests that were planned? I see no reason not to do it. The original results would be misleading otherwise. – Michael R. Chernick Jan 09 '17 at 18:46

2 Answers2

2

As the MCP does not affect the data collection in any way, the only argument against a post-hoc change of the analysis plan would be that you make this decision AFTER looking at the data, i.e. your decision about the MCP might now be (subconsciously) influenced by the knowledge of the outcomes, e.g. the p-values of the three endpoints.

As long as you are sure that this is not the case (a good test would probably be to go to a random statistician, explain only the design, and ask what MCP are necessary), I see no problem with a post-hoc correction of the analysis.

A different question is of course if you need a MCP in the first place. Having several endpoints / tests does not necessarily require an MCP, it depends on question and interpretation.

Florian Hartig
  • 7,979
  • 1
    One of my problems with post hoc is that one can "game" the system and try several MCP's until one gets the desired result. – Chris Barker Jan 10 '17 at 02:28
  • 2
    As I said, if your concern is that you are biased by now, describe your problem in a neutral / blinded way to 1-2 statisticians and let them decide on the most appropriate correction. You may want to keep records of this process in case you need to justify the decision to a reviewer / agency. The process is of course still easier to manipulate than a pre-analysis plan, and one could argue that the very fact that you re-analyze this data has to do with the previous results, but I think it's still better than to stick with an inappropriate analysis (assuming you really need an MCP). – Florian Hartig Jan 10 '17 at 12:49
1

For multiple comparison procedures to control the type I error rate, they should indeed be pre-specified. The only retrospective argument that may hold some water might be that there had been no/insufficient awareness of multiple comparison issues, but that no matter what reasonable multiple comparison procedure one would have chosen the results are still always significant. For a set of reasonable (and pretty conservative) procedures, I might consider a 1/3 : 1/3 : 1/3 Bonferroni split, as well as hierarchical testing in any of the 6 possible orders.

Björn
  • 32,022
  • If you follow through with this logic, if in the protocol of a clinical trial multiple tests are plans on the entire sample of patients and 6 on various subgroups and then in a paper the 7 tests are analyzed separately, it would be acceptable? You can have them announce in the paper that they recognize the importance of multiple testing but they don't have to do it because the individual test that were analyzed were planned in advance. You are thereby allowing several false discoveries to occur. – Michael R. Chernick Jan 09 '17 at 19:43
  • 1
    My point was that I would not take any unadjusted p values too seriously, unless they were so striking that more or less no matter how you adjust for multiplicity it would have all been significant (e.g. every single one <0.0001). – Björn Jan 09 '17 at 21:46
  • I can agree with that. A Bonferroni adjustment which is easy to compute would assure you if that adjustment doesn't make the adjusted p-value large. My concern with the OPs question was that he seemed to presume that it would not be appropriate to do any p-value adjustment if it is done post hoc. What I am suggesting and I think you are too, is that if multiple tests were planned for the study but multiple tests were not implemented.a reviewer is justified to see how multiple testing p-value adjustment affects the conclusions of the study. – Michael R. Chernick Jan 09 '17 at 22:16
  • As I reread your answer and combine it with your reply to my comment, I can see that we would both want post hoc p-value adjustment under the same circumstance. But you are less assertive. Note that you said "The only retrospective argument that may hold watermight be ......" . – Michael R. Chernick Jan 09 '17 at 22:29
  • 1
    Hi, one of the problems I have is that we've already looked at the data and done various new analyses (as yet unpublished). We can in principle "game" the MCP and select one that gives us our desired results. – Chris Barker Jan 10 '17 at 02:31
  • 1
    I was indeed worried about gaming the MCP adjusment, which is why I felt one could not really see the p-values for the three primary endpoints to be significant at the 0.05 level, unless more or less any adjustment would have left them so (I think I am even being more demanding than Michael). I am less clear on how the MCP adjustment for the original primary analyses affects any new proposed analyses. – Björn Jan 10 '17 at 06:04