1

I am new to the site so apologies if this (probably quite straightforward) question has been answered before-

I have carried out research on an intervention with 2 groups (control and intervention), and collected pre and post data for a number of measures (stress, mood, etc.).

To measure the efficacy of the intervention (or lack thereof), I think I need to use a mixed between-within ANOVA to compare the 2 groups on each of these dependent variables.

Would I be correct in saying that? I'm fairly sure, but thought it best to check with those in the know!

I plan to carry out further analyses but this is my main focus, so any help would be much appreciated.

Peter Flom
  • 119,535
  • 36
  • 175
  • 383
Iain
  • 13

2 Answers2

1

I'm betraying my econometrics training by thinking of anova as a special case of regression. But you could treat this as a regression in which each time period is an observation and which you control for variables for your intervention, and for your individual identifier as a factor variable (or "fixed effect"). People trained outside econometrics would probably advocate for "random effects."

If you go the "fixed effects" route, you will want to adjust your standard errors to account for the fact that multiple observations of one person over many periods are not independent of each other. You can think of it like your effective sample size being smaller than the number of people*number of time periods, but greater than the number of people. For this you use a "cluster (or cluster-sandwich)" variance-covariance estimator.

The interpretation of your results depends in large part on how the intervention was assigned. Was it random? Or was the timing random? These sorts of questions will help you figure out whether your regression coefficients have "causal interpretation".

Things that I've "quoted" are things that you could profitably google.

generic_user
  • 13,339
  • People trained outside of economics (say biomedicine or psychology) would have no idea what you are talking about ;-) Note that there are only two time periods (before/after), is all this really relevant or necessary in this case? – Gala May 29 '13 at 05:54
  • Well, if one were to run y = a + bT + e, and y was N2x1, T was N2x1 (being 0 in before and 1 in after), then controlling for ID as a factor variable effectively controls for all time-invariant heterogeneity. It gets at the difference between before and after, confounded only by differential time-variant heterogeneity. Even if treatment was randomly assigned, estimates will have more precision when these "fixed effects" (or individual factor variables) are included. It basically gives everyone their own intercept. – generic_user May 29 '13 at 07:00
1

You can find a lot of relevant material in Best practice when analysing pre-post treatment-control designs As you will see, mixed ANOVA is not actually recommended in this case, ANCOVA is generally considered superior. If you do go for the ANOVA, please make sure that you are looking at the interaction effect, not the main effect for “group”. In any case, as @ACD rightly stressed, interpretation is going to depend a lot on whether or not membership in the control or intervention groups was properly randomized.

One difference between your question and the previous one is that you want to consider multiple dependent variables. This is a problem because multiple tests increase the error rate. One way this is usually dealt with in clinical trials is to designate one dependent variable in advance as the “primary endpoint”. If you fail to find an effect on this measure, you stop there and conclude that your intervention wasn't effective. Other results should only be used to help interpret this endpoint or suggest further research but not to weasel your way out of a failed trial. (That's also one of the thing trial registration could in principle improve; If you commit publicly to a primary endpoint, it's more difficult to convincingly explain away a negative result on this particular endpoint.)

Community
  • 1
Gala
  • 8,501
  • Thanks for all of your help guys- yes, ANCOVA is definitely a more suitable choice in this case! And for the extra details, much appreciated. – Iain May 31 '13 at 10:26