Assessing the Need for a Multilevel model - Random intercept and slope

Question

I am analysing a longitudinal dataset of drug use in R code. Which contain the dependent variable: use measured twice which I want to regress on several independent variables, to see if they can predict changes in use over the two timepoints.

The 3 time-invariant independent variables are age, sex and ´bias´, which is a measure of implicit cognitive bias to the drug. I have a hypothesis that bias can predict changes in drug use.

In assessing the need for a multilevel model I have used the book: Applied longitudinal data analysis by Singer & Willet (avaliable here and : relevant R code). Here the authors suggest that before I regress on predictors I assess "whether there is hope for future analyses" by fiting two unconditional models that: partition and quantify the outcome variation in two important ways: first, across people without regard to time (the unconditional means model), and second, across both people and time (the unconditional growth model).As I understand it, the second model should be a better fit if there is considerable systematic variation in your dependent variable that is worth exploring with predictors.

I fit these models using the R package:nlme like so :

model.a <- lme(use ~ 1, random= ~1|id, data = df)
model.b <‐ lme(use ~ time, random= ~time|id, data= df)

In Singer & Willet's example data their second model.b provides a drop in level-1 residual deviance and their AIC & BIC also drops compared to model.a, indicating a better fit: shown here in their R code.

This is in contrast to my models where AIC & BIC increases

> anova(model.a,model.b)
        Model df      AIC      BIC    logLik   Test    L.Ratio p-value
model.a     1  3 1058.395 1068.320 -526.1975                          
model.b     2  6 1064.356 1084.176 -526.1781 1 vs 2 0.03872928   0.998

As I understand Singer & Willet it is arguable inappropriate to model this using multilevel models, because of the decrease in fit, but I am not sure I understand why.

Question 1: Why is it inappropriate to model this using multilevel modelling?

Question 2: Is the second model not a better fit because I only have two timepoints for the dependent variable in my dataset, and Singer & Willet example has three timepoints?.

The most relevant CV thread I've found is this: Under what conditions should one use multilevel/hierarchical analysis?, But I have not found any answers that satistify this specific case.

Thanks for Reading!

Answer 1

To answer your first question:

Yes, the unconditional growth model should offer a better fit than the unconditional means model, and if not, one should not proceed with further analyses. Here is why.

A multilevel model tests two types of variance: between- and within-person variance. The unconditional means model, in essence, exclusively tests between-person variance.

Each person has a person-centered mean: in your case, it's the average of the participant's drug use across all three time points (let's assume it's three for the time being for explanatory purposes). For instance, if participant 1 used drugs nine times at time point 1 (T1), 12 at time point 2 (T2), and six at time point 3 (T3), her person-centered mean would be 9 (the average of nine, 12, and six).

Now let's say participant 2 had a score of 12 at T1, six at T2, and three at T3. His person-centered mean would be 7 (the average of 12, six, and three). Person-centered means are calculated like so for all participants.

The only purpose of the unconditional means model is to test whether there is sufficient variance in the outcome (i.e., drugs) for participants 1-i (with i denoting the final participant). If your subjects do not vary in this regard, there is no variance to be explained, period, so further tests are useless.

Now we get to the unconditional growth model. The unconditional growth model tests whether there is sufficient variability in the trajectories of change over time among subjects - that is, within-person variance. In other words, do participants 1-i show different evolutions in their attitudes towards drugs?

You found that the unconditional growth model did not offer a better fit than the unconditional means model. Thus, a model constraining each participant's scores to be equal to his or her person-centered mean did not fit more poorly than a model allowing them to differ.

Going back to our example, for Participant 1, the deviation of her T1 score (9) from her person-centered mean (9) was 0. At T2, that deviation would be -3 (9-12); at T3, 3 (9-6).

If the unconditional growth model does not explain additional variance, then these deviations are negligible, and there is no change over time to account for. Thus, Singer and Willett are correct to argue that further analyses are inappropriate because, in essence, there is nothing else to analyze.

To answer your second question, two time points are insufficient for a longitudinal analysis. There is only one difference score, and therefore true change cannot be distinguished from measurement error.