1

Let's say there was an intervention delivered using method A and method B. There was a post test asking participants about satisfaction on a Likert scale immediately after each method was delivered. We want to know which was better (i.e. higher mean satisfaction score).

Let's say 25 schools participated, with 2 individuals per school. One person got intervention A only, and the other got both A and B.

Now, if I only looked at the 25 people who got both interventions, I could probably compare their A & B satisfaction scores using a paired t-test. But that wouldn't work if I want to include observations for folks who only took A since they only have observations in 1 data set, right? There would also have to be adjustments for clustering probably?

Advice on what stat tests/ adjustments are best for this scenario?

There's also the age old question of parametric vs. non-parametric tests for skewed Likerts with somewhat small sample sizes. There's so much conflicting advice out there & simulations seem to show that it's kind of a wash, but if there's something I'm missing please let me know.

Thanks in advance!!

xdrenched
  • 83

1 Answers1

1

So there are 25 schools, 50 students and 75 measurements. Create the data set in following format:

  School  Student  Method  Y    other covariates
     1       1      A      1      ...
     1       2      A      2      ...
     1       2      B      4      ...
     2       3      A      3      ...
   ..................................
    25      50      B      4       ...

Let consider linear mixed model first: $$Y_{ijk}= \beta_0 +\beta_1 X_1 + \text{ other covariates } + \gamma_{1i} +\gamma_{2j} +\epsilon_{ijk}$$ where $i$ indicates the school, $j$ student, $k$ the order of measurement. $X_1 = 1$ for methods B and = 0 for method A. $\gamma_{1j}$ is random school specific intercept, and $\gamma_{2j}$ is random school specific intercept, $\epsilon_{ijk}$ is error term. All of them following the normal distribution with mean 0 and variances $\sigma_1^2$, $\sigma_2^2$, and $\sigma_3^2$, and they are independent from each other.

Based on this structure, the variance-covariance of 3 $Y$s from the same school is: $$Cov(Y_{111}, Y_{121}, Y_{122})=\left(\begin{matrix} \sigma_1^2+\sigma_2^2+\sigma_3^2& \sigma_1^2 &\sigma_1^2\\ \sigma_1^2 &\sigma_1^2+\sigma_2^2+\sigma_3^2& \sigma_1^2+\sigma_2^2\\ \sigma_1^2&\sigma_1^2+\sigma_2^2&\sigma_1^2+\sigma_2^2+\sigma_3^2\end{matrix}\right) $$ It indicates that relation between $Y$ from the same student is stronger than that from different students from the same school. $Y$s from the different schools are independent. In this model $\beta_1$ is the difference between method A and B, and you can estimate and test it.

If you think the linear model is not good, especially $y$ does not follow normal distribution, you can select other distribution and/or link function and keep the fixed and random effects parts not change. $$\beta_0 +\beta_1 X_1 + \text{ other covariates } + \gamma_{1i} +\gamma_{2j}$$

user158565
  • 7,461
  • Thanks!! This is really helpful thorough answer - especially showing the data structure. Sorry for my ignorance - is this getting into instrumental variable territory? So the bottom line is that I basically have to do some kind of multivariable analysis/ regression, right? Bivariate is out of the question since I now have IVs for intervention type, and control variables for school and individual too? I anticipate the DV will be pretty skewed, so maybe dichotomizing the Likert or scale & doing logistic would be better? – xdrenched Nov 21 '18 at 06:47
  • 1
    No instrument variable. If you do not have other covariates, just a simple linear regression with random effects. I have no idea what bivariate means, we just have one DV test score. As I said in the Answer, you can change the distribution and link function to fit any kind of generalized model, but just keek the last line of specifying the fixed effect and random effect. – user158565 Nov 21 '18 at 14:58