0

I am trying to evaluate the effect of a certain policy on the achievement gains of students between two points in time. However, not all the students are exposed to this policy. I am supposed to use the gain score as a dependent variable.

After having read many critical articles about the use of gain scores as outcome variables, I would like to ask if somebody could tell me the advantages of this approach (compared to using the post-test score as an outcome variable while controlling for the pre-test score).

Is it wrong to say that only when using the difference score as an outcome I can really answer the question of whether competences of students grow faster when they are exposed to this policy? I feel like using the post-test score as an outcome variable does not precisely give me an answer to this question.

I would really appreciate if somebody could give me some advice on that

kjetil b halvorsen
  • 77,844
Econometrica
  • 1

1 Answers1

1

The issue you are describing is well understood in the context of clinical trials, see for example Chapter 7.2.3 in Senn (2008).

You may think that comparing difference scores (also known as change scores) and comparing post-test scores while adjusting for pre-test scores (also known as analysis of covariance, ANCOVA) are different, however, the former is actually a special case of the latter.

To see this, denote by $Y_{1i}$ and $Y_{2i}$ the pre- and post-test score of student $i$, respectively. Whether student $i$ received the policy is coded with a dummy variable $X_i$, which is 1 if they received it and 0 otherwise. The ANCOVA model is now given by $$ Y_{2i} = \mu + \beta X_i + \gamma Y_{1i} + \epsilon_i, $$ where $\mu$ is the average score when receiving no policy, $\beta$ the effect of the policy, $\gamma$ the influence of the pre-test score on the post-test score, and $\epsilon_i$ a random error.

The change-score model, on the other hand, is given by $$ Y_{2i} - Y_{1i} = \mu + \beta X_i + \epsilon_i, $$ which can be identified as a special case of the ANCOVA model with $\gamma = 1$ and $Y_{1i}$ subtracted from both sides. In contrast, if we set $\gamma = 0$, we obtain a model where the post-test score does not depend on the pre-test score at all, the reality is probably somewhere in between.

What this means in practice is that the change-score model is a more restrictive model compared to the ANCOVA model. It may well be the case that the influence of the pre-test score is not equal to one, which in turn will reduce the efficiency of your estimate of $\beta$.

SamuelPawel
  • 81