1

I have a model that looks into a 3-way interaction between 2 categorical variables (Group: Gr. A, Gr. B, Gr. C; Area: A1, A2, A3) and a numerical linear interaction (Distance from center - ECC: 1, 2, 3) on participant accuracy. Formula = lmer(Accuracy ~ Group + ROI + Group:ECC:ROI + (1+ECC|ID), data =Ecc, REML = TRUE).

The effects of Group and ROI behave as expected (i.e. they are reported relative to the reference level, i.e. slope change relative to the Accuracy values of Group A and Area A1). However, the interaction effect seems to be returned relative to a straight line (i.e. slope change relative to a no slope - straight line - ). The output returns the interaction of the reference level as well as others, and is not affected by changing the reference level.

Can anyone explain why this is the case, and would you ahve any references to read more in it? We couldn't find much....

Here is the result of our model using the report (https://www.rdocumentation.org/packages/report/versions/0.5.7/topics/report) function:

We fitted a linear mixed model (estimated using REML and nloptwrap optimizer) to predict Accuracy with Group, ROI and ECC (formula: Accuracy ~ Group + ROI + Group:ECC:ROI). The model included ECC as random effects (formula: ~1 + ECC | ID). The model's total explanatory power is substantial (conditional R2 = 0.43) and the part related to the fixed effects alone (marginal R2) is of 0.19. The model's intercept, corresponding to Group = A, ROI = A1 and ECC = 0, is at 32.38 (95% CI [26.20, 38.55], t(333) = 10.32, p < .001). Within this model:

  • The effect of Group [B] is statistically non-significant and positive (beta = 9.13, 95% CI [-0.71, 18.97], t(333) = 1.83, p = 0.069; Std. beta = 0.99, 95% CI [0.62, 1.36])
  • The effect of Group [C] is statistically non-significant and positive (beta = 3.37, 95% CI [-5.73, 12.47], t(333) = 0.73, p = 0.467; Std. beta = 0.33, 95% CI [-7.09e-03, 0.67])
  • The effect of ROI [A2] is statistically non-significant and negative (beta = -3.28, 95% CI [-9.13, 2.57], t(333) = -1.10, p = 0.271; Std. beta = -0.15, 95% CI [-0.35, 0.06])
  • The effect of ROI [A3] is statistically non-significant and positive (beta = 4.10, 95% CI [-1.75, 9.95], t(333) = 1.38, p = 0.169; Std. beta = -0.13, 95% CI [-0.34, 0.07]) **- The effect of GroupA × ROIA1 × ECC is statistically non-significant and positive (beta = 2.95, 95% CI [-1.48, 7.39], t(333) = 1.31, p = 0.191; Std. beta = 0.14, 95% CI [-0.08, 0.35])
  • The effect of Group [B] × ROIA1 × ECC is statistically significant and positive (beta = 11.55, 95% CI [4.85, 18.26], t(333) = 3.39, p < .001; Std. beta = 0.50, 95% CI [0.14, 0.85])
  • The effect of Group [C] × ROIA1 × ECC is statistically non-significant and positive (beta = 4.55, 95% CI [-1.51, 10.62], t(333) = 1.48, p = 0.141; Std. beta = 0.23, 95% CI [-0.09, 0.55])
  • The effect of GroupA × ROI [A2] × ECC is statistically non-significant and positive (beta = 2.38, 95% CI [-2.05, 6.82], t(333) = 1.06, p = 0.292; Std. beta = 0.16, 95% CI [-0.06, 0.38])
  • The effect of Group [B] × ROI [A2] × ECC is statistically significant and positive (beta = 12.69, 95% CI [5.98, 19.39], t(333) = 3.72, p < .001; Std. beta = 0.57, 95% CI [0.21, 0.92])
  • The effect of Group [C] × ROI [A2] × ECC is statistically significant and positive (beta = 7.35, 95% CI [1.29, 13.42], t(333) = 2.38, p = 0.018; Std. beta = 0.23, 95% CI [-0.09, 0.55])
  • The effect of GroupA × ROI [A3] × ECC is statistically non-significant and negative (beta = -3.15, 95% CI [-7.59, 1.28], t(333) = -1.40, p = 0.163; Std. beta = -0.20, 95% CI [-0.42, 0.02])
  • The effect of Group [B] × ROI [A3] × ECC is statistically non-significant and positive (beta = 3.89, 95% CI [-2.82, 10.59], t(333) = 1.14, p = 0.255; Std. beta = 0.21, 95% CI [-0.14, 0.57])
  • The effect of Group [C] × ROI [A3] × ECC is statistically non-significant and negative (beta = -1.90, 95% CI [-7.97, 4.16], t(333) = -0.62, p = 0.537; Std. beta = 2.27e-10, 95% CI [-0.32, 0.32])**

Standardized parameters were obtained by fitting the model on a standardized version of the dataset. 95% Confidence Intervals (CIs) and p-values were computed using a Wald t-distribution approximation.

Thanks in advance!!! ############################################################################

Only relevant in the context of discussion comments below: Illustration of ME following discussion with dipetkov, at the moment we're finding non-sifniciant differences, I'd expect to find significant differences between group A and B, especially when referencing to A3, Group A

All effects > thanks for the recommendation caroZ

I'd specially expect significant difference here bars = observed, blakc lines and x =predicted from model

Malen
  • 11
  • 2
    A list of coefficient estimates, significant or not, is hard to read & interpret. A more fruitful, or at least more visual, approach would be to plot predicted accuracy for different combinations of group, ROI and ECC. See for example these tutorials: 1 with marginaleffects, 2 with emmeans. – dipetkov Nov 13 '23 at 18:42
  • 1
    PS: The model doesn't include a main effect for ECC nor two-way interactions. Why? – dipetkov Nov 13 '23 at 18:43
  • 1
    To understand the basis for the comment (+1) from @dipetkov see this page for a general explanation of why you need to include all main effects and 2-way interactions, and this page on a specific 3-way interaction model. Three-way interactions are hard enough to understand even when all lower-level terms are included; it's almost impossible to think clearly about 3-way interactions when lower-level terms are omitted. – EdM Nov 13 '23 at 19:15
  • Sorry maybe I missed the information, but what is the stat of your full model with the interaction in ? Then I would personally use the

    package and do
    – CaroZ Nov 13 '23 at 23:10
  • @dipetkov, the main effect of ECC is not included in the model because we were getting the following warning when doing so: fixed-effect model matrix is rank deficient so dropping 1 column / coefficient . This issue was not the case when ECC was not included as ME. I thought that this was the case because the modelw as computing GroupAxA1xECC twice, once by calculating the ME on the reference level (Group A, A1), and another by calculating the interaction with Group A, A1, ECC. – Malen Nov 14 '23 at 12:59
  • @dipetkov, in relation to the 2-way interaction, do you think I should also include Group:area, Group:ECC, Area:Ecc as well? We did not include them because what we're really itnerested is the interaction between the 3, but if this is malpractice we can inlcude them also! – Malen Nov 14 '23 at 13:01
  • The warning about a rank-deficient fixed-effect matrix means that there is either some problem in the coding of the categorical variables or perhaps some combination of categorical-variable values that is missing from the data. You need to address that issue first. If ECC only takes on values of 1, 2 and 3 (as I understand the question) then it's possible that it's also being treated as a categorical predictor. – EdM Nov 14 '23 at 14:33
  • 1
    I would include all terms: Acc ~ Group * ROI * ECC. The two categorical terms, Group * ROI, effectively define 9 subgroups. So the three way interaction means each group would have its own (fixed-effect) intercept and ECC slope. The individual intercept / slopes would vary about those group-specific ones. – dipetkov Nov 14 '23 at 14:41
  • 1
    About errors during the fitting: As @EdM points out it can be about how the variables are coded. Or perhaps the dataset is too small for the 3-way interaction model. It would help to make plots; they can help to check details such as the coding but also to gain understanding of the relationships between the variables observed in the data. (Also, graphs help to get more specific feedback on CV when added to the question.) – dipetkov Nov 14 '23 at 14:46
  • mmm I see, so I ran the model you recommended but the 3-way interactions are not making as much sense now... especially is saying that there is no difference in slopes between the reference level (green bars) and comparison level (in this case Area 2 from group B) -- adding in the post -- the bars are the observed means, the black lines and xs are the predicted means by the model – Malen Nov 14 '23 at 16:12
  • Thanks for the graphs. I'm not sure what you mean by "non-significant differences" between groups A and B. Visually, the fitted lines are different. I suspect you may be looking at the significance of the individual model coefficients? One final comment, as I just noticed this: the outcome is accuracy. Are the original observations of the participants binary (eg. success or failure, coded as 0 and 1)? If yes, then it would be more appropriate to use a GLM / lme4::glmer. – dipetkov Nov 14 '23 at 21:24
  • Hi @dipetkov, yes that's what I mean, from my understanding the coefficients can tell you how different the slope of the comparison level (e.g. Group B:ECC:Area B) is from the one of the reference level (Group A: ECC:Area A) (?), is there a better way to asses this from your point of view? Accuracy here refers to a mean classifier's performance, so it ranges from 0:100% – Malen Nov 15 '23 at 13:52
  • Take a look at the links in the first couple of comments. The challenge of interpreting a regression is that it's not necessary that every single model coefficient is "statistically significant" for there to be important and scientifically interesting differences between various subgroups. – dipetkov Nov 15 '23 at 14:09

0 Answers0