3

I'm having an argument with a colleague (neither of us have a strong statistical / research methods background) on whether we should randomize questionnaire layout.

The survey in question is a feedback forms on teacher performance which are distributed for high school pupils. So one pupil gets up to 20 surveys (on each teacher he has contact with) which have identical, multiple-choice questions. Most of the questions also have identical scale. The pupil fills the surveys in online and does not necessarily need to do everything at once (but he definitely can). We do not have control how / when they choose to do it.

My colleague argues that by randomizing the order of questions we reduce the possibility that pupils will fill in the questionnaires without thinking too much - e.g. if one fills 5 forms at once, the evaluations of teachers will be very similar.

In my opinion, though, we would actually gain from making sure that the survey questions are grouped together based on the area of question (e.g. teacher communication skills, usage of IT, etc) - this would ensure that pupils provide consistent answers.

I guess there's a point in both approaches - but maybe there are some best practices how to deal with that?

Oh, and if somebody could dig up some references, that would be ultra-cool. My colleague likes facts :)

Aurimas
  • 133

1 Answers1

5

The argument for randomising the order of the questions is due to the fact that the answers to later questions can be biased by the presentation of earlier questions. Randomising the question order means that the influence is no longer subject to this ordering bias.

With repeated administration of the same survey, the ordering effect is only removed for the first time the person completes the questionnaire. I would randomise for the first survey, and then not worry about it for the following surveys.

This concern has been around for decades, I imagine the main paper came out in the 1950s or 1960s. But here are some references to look at:

Any good questionnaire design book will cover this off.

Michelle
  • 3,900
  • thanks for the answer. I understand this deals with the issue that, as you say "answers to later questions can be biased by the presentation of earlier questions". But what about the fact that a respondent is filling in multiple identical questionnaires on different teachers? Our main concern is not "bias within a questionnaire", but rather "bias accross responses on different teachers from a same pupil". – Aurimas Mar 15 '12 at 08:59
  • The same principle would apply: randomise the sequence of the teacher order across the students. – Michelle Mar 15 '12 at 09:22
  • yes, but the bias comes from the fact that the questionnaires are identical - so would it not make more sense to randomize the order of questions in the questionnaire (differently for each teacher), so that they are no longer identical? Sorry, if that feels like me pushing further - I just have a feeling I might have not explained the issue well enough. – Aurimas Mar 15 '12 at 11:57
  • Once the students have seen the questionnaire once, it makes no sense to randomise the order of the questions, because they already know what the questions are. So you can only fix the order bias in the first question. My comment above suggests how you can prevent teacher order bias. – Michelle Mar 15 '12 at 16:08