No significant interaction in lmer - do you refit the model without interaction?

Question

I'm running/ reporting lmer analysis for the first time and have a hypothetical question related to a few of my models. I'm using R lmer package - my models are generally of the formula:

model=lmer(dependantVar ~ continuousVar*catagoricalVar+  (1|Subject), data=data, REML = FALSE);
anova(model)

In some instances, I have an interaction (sig p value) - this is fairly straightforward as I interpret that and "ignore" the main effects.

However, when I have no interaction, but a main effect of the continousVar only. I'm not sure what to do in terms of reporting the findings. Do I report the main effects estimates from the model above, or do I refit the model without the interaction terms first? i.e.

 modelreduced=lmer(dependantVar ~ continuousVar +  (1|Subject), data=data, REML = FALSE);

or

     modelreduced2=lmer(dependantVar ~ continuousVar + catagoricalVar +  (1|Subject), data=data, REML = FALSE);

and report these estimates.

It's not as if they change a great deal but I'm trying to gage from the literature what people do, and there seems to be examples of both. Although I feel like leaving the interaction in is incorrect, I have come across instances of it being reported.

Any pointers to the "proper way" (if there is one) would be appreciated.

I should say that my main interest is my continuousVar on my dependantVar, but I did believe catagoricalVar may effect the steepness slopes, even by a small amount, so wanted to capture this in the model. So even if it's not significant (if we defer to p values here) should I leave it in anyway?

Also to note, whilst there is a slight reduction in AIC values:

lrtest(model,modelreduced)
lrtest(model,modelreduced2)

suggests no significant difference in these models.

Thank you.

Answer 1

Chapter 4 of Frank Harrell's Regression Modeling Strategies covers that and many other issues relevant here. The general strategy is to decide on how many coefficient values you can try to estimate without overfitting ("degrees of freedom"), decide how many degrees of freedom to spend on each predictor (including interactions), then spend those degrees of freedom in the model and report the results.

Thus it's best to decide beforehand on whether to include an interaction, based on your understanding of the subject matter. If you decide to include an interaction, then report the model with the interaction. If you later refit the model based on the "significance" results of the interaction term, then you've used the outcome to choose the (new) model in a way that violates the assumptions behind significance testing.

There is, however, another place where you might want to spend more degrees of freedom: your modeling of continuousVar. As Harrell explains, a strictly linear association of a continuous predictor with outcome is unlikely to hold in practice. A more flexible model, like a regression spline, might be called for; you would include an interaction of the spline with categoricalVar. That would be wise, if your data set is large enough to allow you to spend more degrees of freedom. Again, you would pre-specify the model with the spline and report the results, not re-fitting even if the nonlinearity is "not statistically significant."