6

I am doing research on students and their perception of their grades. Specifically, I want to do an experiment where students either (a) see their actual grades in a course (as a percentage) - the control condition or (b) their actual grades in a course (as a percentage) as well as their percentile in the class - the treatment condition. For example, a student could be earning a 90% in the class (as a percentage) but be in the 99th percentile for their class. The key question of interest is whether seeing the percentile information influences students motivation etc.

However, I was wondering about treatment interference. That is, the percentile information in the treatment condition is dependent. If one student is in a higher percentile, that necessarily means that another student in the class is in a lower percentile. With this in mind, is my experiment valid? Is there any ways I can get around this problem? I want to be sure I'm drawing the correct causal interpretation.

user2917781
  • 515

1 Answers1

1

I would say this is indeed valid.

What you likely measure by comparing the control group

  • those who only see their actual grades in a course (as a percentage)

with the treatment group

  • who see both their actual grades in a course (as a percentage) and their percentile rank

is essentially the effect of presenting a (relative) measure in which your relative gain means someone else's loss.

Statistically, you should be aware of two other problems with ranks (see also here):

The first is ranking changes with respect to sample size: In small classes, a minor change in the actual grade could lead to substantial percentile rank changes (e.g. from 60% to 70% if there are only 11 students in the class). If treatment and control group class sizes differ, this could in some sense bias your interpretation (depending on the mechanism you are expecting). Usually, you would more seldom use a ranking much in very small class (e.g. for data privacy reasons). Presenting a rank among a group of let's say five people could let students act differently for other reasons than you have in mind I think.

The second is the shape of the underlying distribution. A rank change of one percentile pont can be easier or more difficult depending on where you are. Think there is one extreme hard task in the exam while all others are easy. You can easily climb to the 80th percentile but another rank increases requires to have an idea how to solve that task.

Arne Jonas Warnke
  • 3,513