Progression between two exams under three conditions

Question

I want to test if three types of learning mehods have different effects on the progression between two exams.

The data was collected like this:

• A first exam was done with all participants. They had various level of knowledge of the subject, ranging from expert to novice.

• The participants were randomly separated into three groups which were given a lecture based on text, pictures or animations.

• They all passed a second exam (the same for everyone, but not the same as the first one) on the same subject.

I would like to know which learning method was the best one in that case. I was thinking of calculating the "progression" like this: $$ \text{score on the second test} - \text{score on the first test} $$ And to run an ANOVA on this. However I realized that participants with very good previous knowledge had a low progression because they scored high on both tests while participant with very low previous knowledge had a much better progression even if the score at the second test was below average.

I thought of removing the participants with high scores on the first test, but I am not sure it is a very good solution. I also thought of doing something with the ranking: average the ranking of the user under each condition before and after the lecture and see if one condition allowed a greater increase in average ranking.

What do you think?

Answer 1

Using a basic ANOVA on change-scores is probably a bad idea with an experimental factor. You probably want a mixed-effect ANOVA if you're looking to falsify the null-hypothesis that all three groups' scores changed equally. If you just want to know how different the means are and don't care about quantifying the strength of your evidence against the null, you might just want to estimate effect-size and skip the significance-test. You may have some difficulty meeting the assumptions of a parametric ANOVA, especially if several participants are scoring 100% or otherwise producing negatively skewed distributions. Sphericity is a particularly tricky assumption to meet. Therefore if you do want a significance test, consider looking up nonparametric or otherwise robust methods. If your participants are really hitting the maximum score a lot, you may want to try a method designed for right censored distributions.

Answer 2

There are two issues here: (1) what to do about the fact that you have repeated measures on each participant and (2) what to do about the fact that the scores have a limited possible range.

1. If you have more that two scores per person, life gets more complicated, since you have only 2, there are simpler options available. With >2, you would need to use some form of mixed effects model, such as the repeated measures ANOVA that @NickStauner mentions. You can still use that here, but you can just as well go with a simpler model, which I suspect you might prefer. With just two measurements there are two strategies that people often use. You can subtract the earlier from the latter to create change scores (that's what you were trying), or you can use the earlier score as a covariate in a multiple-variable model. How to choose between these options has generated a lot of controversies over the years (see here: Best practices when analyzing pre-post treatment designs), but a quick version of the conventional wisdom is that you should use change scores if you cannot assume the groups were equal on average beforehand, but should prefer controlling for the earlier score as a covariate otherwise because of the advantages this approach offers. Specifically, you can see the effect of having a higher or lower initial score. Since you randomized your participants, you can use the covariate approach.

2. The limited nature of your data poses a much trickier problem. There are very sophisticated ways of modeling such data in psychometrics using item response theory, however, I doubt you have enough data and you don't want to have to try to learn IRT overnight if you aren't already familiar with it. My guess is that the easiest option for you is to use an ordinal logistic regression model predicting score2 from group and score1. You may also need to include a squared term for score1 to capture a possible curvilinear relationship. If you have enough data you could also include interaction terms to assess if some instructional methods are better for students who are already doing well / poorly, should that be a question you are interested in.