5

I have the results of a 2*5 mixed ANOVA drug trial, where there’s a non significant interaction but main effects of both time and group. Scores were taken before and after trial (time). 4 drugs were tested and 1 control.

How do I interpret this non-interaction and main effects? Does the interaction mean that there was no difference between groups on the time measure?

Also, is the control group throwing off the whole study? If there's no interaction does a higher control mean no drugs have any effect? graph

Harry
  • 51
  • 3
  • Welcome to CV! Could you plot your score as a function of time and group (either from raw data or from model results), and edit the plot and your ANOVA output into your question? – Sointu Feb 17 '24 at 07:45
  • I didn't mean the data itself but the model output and a results plot. But even if you can't share them, looking at the visualization will probably help you see a lot more about what's going on in the data. But with the information available, yes it sounds like there is no effect of drug on the response score and the main effects are because one of the groups had higher scores to begin with, and time itself had some effect. – Sointu Feb 17 '24 at 07:56
  • The lines can be non-parallel even if there's no statistically significant interaction. And you don't need to do post hocs when there's no interaction, but you can, if these post hoc are planned. But it's a bit hard to say what's going on without seeing the results. – Sointu Feb 17 '24 at 08:08
  • Yes, thank you. It does seem that you couldn't reliably say that any of the drugs have an effect based on these results. Scores decrease in all groups and differences in time effects are small to nonexistent. – Sointu Feb 17 '24 at 08:25
  • Thank you Sointu! – Harry Feb 17 '24 at 08:26
  • 1
    Yes, because when you choose to use mixed ANOVA to investigate a drug effect in a pre-post design, the interaction between time and group is your critical effect, not the main effects. – Sointu Feb 17 '24 at 08:35
  • 1
    But if you have a mixed ANOVA with time included, the main effects of group from that model are overall effects (averaged over both time points) so they don't tell you whether changes over time differ between groups, which is what would tell you whether the drug worked. – Sointu Feb 17 '24 at 08:40
  • Treatment effects can also be studied using ANCOVA with the baseline scores as a continuous covariate and the group as the between-subject factor without any repeated or within-subject element. In this design the group main effect is your critical effect. So you could try that but then again if you change your analysis because you didn't get significant results from you first one, that can be considered p-hacking (fishing for results, which makes any results found less reliable). Then again, maybe ANCOVA would have been the correct analysis for you in the first place. – Sointu Feb 17 '24 at 08:41
  • OK sorry, I thought you meant post-hocs for just the group. So generally yes, you can run post-hocs for the interaction contrasts even if the interaction itself is not significant and consider the significant post-hoc results, results. However, I'm not sure whether this is appropriate in drug trials, I've always been under the understanding that time x group interaction is critical in those. But I notice now I'm not sure - I hope someone with bio science experience steps in! Good luck! – Sointu Feb 17 '24 at 08:59
  • (With post-hoc for interaction contrasts in the above comment I mean comparisons between time 1 and time 2 within each experimental group. Those are probably informative. Post-hocs for just group1 vs. group 2 etc. are not informative in the sense you want) – Sointu Feb 17 '24 at 09:06
  • Thanks Sointu. Why would the group main effect not be informative? (Sorry, I’m quite new here!) – Harry Feb 17 '24 at 09:13
  • It is informative regarding whether there are differences between the groups in general (across the two time points). But that's not what you are interested in, right? You want to know whether the drugs worked. Group main effect is not informative regarding the effects of drugs because it averages over the time element. It just tells you e.g. that drug1 group had higher score than control group overall (averaged over pretest and at posttest scores). That's why the time x group interaction is crucial for finding out whether drug had any effect. – Sointu Feb 17 '24 at 09:16
  • In mixed ANOVA you are using each participant as their own control, so if you randomized your participants, the interaction results should still be valid even if the groups differ at baseline. As I mentioned, another way to overcome the issue is to use an ANCOVA with baseline scores as covariate and group as between-person factor. – Sointu Feb 17 '24 at 09:38
  • 1
    @Sointu "In mixed ANOVA you are using each participant as their own control, so if you randomized your participants..." I agree with this but I think mixed ANOVA makes sense for cross over trials only. That is, when the same patients receives more than one treatment during the experiment. Then that raises the question of how the participants are randomized. The OP doesn't explicitly mention that the trial is a cross over, so I'm curious why they fitted a mixed ANOVA to begin with. – dipetkov Feb 17 '24 at 16:40
  • 2
    Please do not vandalize your question. When you posted on SE, you gave up ownership of the content under CC BY-SA 4.0. If there are no answers, you may delete your own question (see here ): just click the faint gray 'delete' at lower left (your account needs to be registered for this). Otherwise, the thread will remain according to SE's rules. – kjetil b halvorsen Feb 18 '24 at 16:38

1 Answers1

3

Your study objective is to assess effects of four drugs on a health outcome with reference to the effect of a control substance on a health outcome. It is not about influence of time on drug or drug on time. Both drug and time are predictors not the response. An effect is of a predictor on the response.

The main effect of time is negative, shown by the negative slopes of lines from Time 1 to Time 2. It means the health outcome decreased over time no matter which of the five medications to use. The main effects of four treatment-group indicators measure the difference between a treatment group and the control group at Time 1 before using any medication. They are all negative if the control group had the highest score to begin with.

If group assignment is completely random, we expect that the main effects of treatment group to be zero. Significant main effects of treatment group means systematic differences at the starting line, nonrandom group assignment, and a sample-selection bias that undermines causal inference. For sample-selection remedy in causal inference, see my answer at Seeking Assistance in Evaluating My Research Plan for Regression Analysis. You can also use measurements at Time 1 as a predictor instead of a response, to adjust for baseline differences. See Senn, S. (2006). Change from baseline and analysis of covariance revisited. Statistics in Medicine, 25(24), 4334–4344. https://doi.org/10.1002/sim.2682.

You are using the difference-in-difference design, except that you do it four times. For the interpretation of interaction effects in this design, see my answer at Frank Harrell's interpretation of interaction in regression results. In your case, nonsignificant interaction between time and treatment means that the effects of 4 tested drugs on the health outcome were no different from that of the control substance, shown by the parallelism among the five downward lines. It does not mean that the tested drugs have no effects but that the tested drugs have no additional effects that the control substance cannot reach. This is a collective F test. To compare each drug to the control, we need to use t tests on individual coefficients of the four interaction terms in a regression table.

If you are conducting noninferiority study to demonstrate that new drugs are no worse than the current standard, nonsignificant interaction is a desirable result. Note that nonsignificant difference does not necessarily mean noninferiority or equivalence. For the definition and determination, see U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research, & Center for Biologics Evaluation and Research. (2016). Non-inferiority clinical trials to establish effectiveness: Guidance for industry. https://www.fda.gov/media/78504/download

DrJerryTAO
  • 1,514
  • With five groups, you will have four group indicators. The treatment effect in causal inference has a specific definition, distinct from efficacy of a drug. See my answer. If all four new drugs have the same efficacy as the control, the treatment effect is zero. – DrJerryTAO Feb 17 '24 at 18:59