0

I have 290 clusters made up by subjects observed from one to 14 times, and a relative large number (45) are observed only once. I worked in R, and I considered subjects ID as random effect. The first model I fitted is made up of fixed effects (I have 3 $\beta_s$, that is the intercept and 2 regression coefficients) and 1 random intercept. Since I'd make model comparisons based on AIC, by fitting also more complex models with random slope, I estimate it by ML. Is that okay? I do not think ML is a problem since the sample size is large (more than 2000 observations) and the bias in the variance estimation should be negligible (indeed, it is).

The choice of going with a more complex model, I mean with a random slope as well, shall be first driven by theory, right? Or shall I consider a priori a more complex model in light of the (peraphs) higher variance that an additional random effect could explain. Indeed, this is the case. Considering the random intercept as the baseline simplest model, if I fit a model with both random slope and random intercept, it turns out that the variance explained by the random slope is 4 times the residual variance, and twice the variance explained by the random intercept. Indeed, this result occurs only by fitting a random intercept related to one specific covariate out of two. For the other covariate this does not occur, and the variance explained by the random slope is relatively tiny with respect to the variance of the residuals and the variance of the random intercept. Actually, this model (only the one where the variance of the random slope is relatively tiny)cannot be estimated with ML since it fails to converge, and I'm forced to use REML. Is this approach for model selection valid ?

Maximilian
  • 237
  • Hi! About model selection, the answer is No, REML is not valid; you have to use ML. This is explained in the reference I provided you in your previous post here. – utobi Jan 06 '23 at 11:17
  • It is clear that REML cannot allow me to make comparisons based on BIC or AIC. Let say I go with ML. My question is: Is it appropriate to choose a more complex model based on AIC differences and looking at the relative variance explained by the random effect? And another point that I've implicitly highlighted is that many clusters have only one observation. Is that a problem? – Maximilian Jan 06 '23 at 11:21
  • yes, looking at AIC is ok. – utobi Jan 06 '23 at 11:23
  • If the majority of clusters have a single observation perhaps then is better to average out the others and take classical regression. Or at least I'd check both. – utobi Jan 06 '23 at 11:25
  • Not the majority of clusters have a single observation (it is about 16% of them). Thanks for your comments – Maximilian Jan 06 '23 at 11:27
  • 1
    For fitting the model it should not be a problem to have a number of subjects observed only once. The problem is that these subjects cannot be used to check model assumptions regarding the random effects, but this is certainly not a reason to exclude them. With 16% from my intuition I wouldn't expect big trouble. – Christian Hennig Jan 06 '23 at 12:15
  • Thanks very much. Do you recommend to take a sub sample with subjects that are measured at least at two different occasions, and see what happens to the variance covariance matrix, and so to the estimated variance of the random effects, and see also how does the likelihood changes. I’d just temporarily exclude them, for model selection.Still, for model selection, the choice between a model with random intercept vs model with both random intercept and slope, should I rely only on AIC( since the Likelihood ratio test I do not think is appropriate since I do not have a nested model)? – Maximilian Jan 06 '23 at 13:11
  • 1
    I don't see any benefit of removing the subjects with only one measurement. I won't comment on how to do model selection as for me this always involves subject matter considerations and looking at the data. – Christian Hennig Jan 06 '23 at 13:59
  • Thanks again for your comments. I meant to temporarly drop subjects observed just once, to properly look at the relative variance of random effects (relative to the residuals' variance), and then to fit the model with all the clusters. Just last remark, if I may ask (you can also provide an answer so that I might click for the best), just as a general prospective, might model selection work with mixed models (random intercept vs random intercept + random slope), just based on AIC? – Maximilian Jan 06 '23 at 14:07
  • 1
    AIC is OK as a general approach for model selection and I have no particular objections against looking at it in this situation. However even what "model selection works" actually means depends on details of the specific situation that I don't know, so I won't give recommendations. – Christian Hennig Jan 06 '23 at 17:38

0 Answers0