13

This is perhaps the question that some thought Grading scale: how to handle multiple choice questions with different number of choices really was. I'm of the opinion that multiple choice questions should not be used for summative assessment. However, I have no actual evidence to back up that view.

Has there been actual research on this, specifically with regards to mathematics?

I don't know if the use of multiple choice questions is different in mathematics to other subjects by dint of sheer ignorance in how other subjects are examined. I do know that often multiple choice questions in mathematics are not simply "Do you happen to know the answer?" questions where the route to the answer is irrelevant but the goal is more that by seeing the choice of answer one can deduce the route that they took and so there is no need to see the details (this assumes that they took a route and did not simply guess).

A sub-issue in this is the numerous schemes to discourage guessing. I'd happily also learn about any research as to the effectiveness of these. Again, my opinion is that they do not correct for the failings of the use of multiple choice questions but, again, I have no research-based evidence for this.

Let me conclude by re-emphasising that I am asking about summative assessment and not formative assessment and about actual research. I'm not interested in answers that are purely anecdotal or opinion-based (I'll happily hear those in another venue, though).

In thinking about MattF's comment about correlation, let me try to focus it more precisely. I have no doubt that multiple choice test scores are correlated with every other type of test. That doesn't speak to their efficacy and fairness though.

Consider the following two types of question:

  1. Multiple choice, where the options include "Some of the above" and "All of the above".

  2. No partial credit, but credit only given for an answer with reasoning.

In both, it is an "all or nothing". I've posited the extra options for the "multiple choice" variant to make it so that it isn't just a closed list to choose between and so to know the answer then a student ought to have worked it out. In the first, a student can guess. In the second, they can't.

My question, then, could be phrased as: how much effect does the fact that students can guess an answer have on a multiple choice test's ability to report students' abilities when compared with a "no partial credit" test?

Community
  • 1
Andrew Stacey
  • 3,634
  • 20
  • 32
  • Effective at what? Schools admit students based on SATs, ACTs and GREs. States license accountants, architects and engineers based on multiple-choice exams. Financial organizations like CFA and FINRA certify based on multiple-choice exams. The broad use persuades me that multiple-choice questions can assess mathematical learning in a way that is effective for the various purposes of these organizations. What more effectiveness are you looking for, and what research could demonstrate it? –  May 28 '14 at 12:24
  • 3
    @MattF. Effectiveness in assessing what the students have actually learnt. I would not be surprised to learn that all of the situations you cite use multiple choice exams because they are cheap to grade, not because they actually measure what they want to measure. – Andrew Stacey May 28 '14 at 12:33
  • 1
    I could also cite correlations between multiple-choice and long-form sections of these tests, but I doubt that's what you're after. How do you propose to measure "what the students have actually learnt" so as to make this a proper study of research? –  May 28 '14 at 12:44
  • 2
    @MattF. Please cite those. I'll only be able to assess if that's what I'm after if I see them. – Andrew Stacey May 28 '14 at 13:04
  • @AndrewStacey: a definition for the terms "summative" and "formative" for those less familiar may be useful in your question. – Willie Wong May 28 '14 at 14:03
  • 2
    As a grade school student, I tended to treat multiple-choice exams like a game, particularly because mathematics competitions in those grades tended to be multiple-choice. – Joe Z. May 28 '14 at 14:51
  • 1
    Also, the penalty for guessing that standardized exams isn't there to discourage guessing, but to provide leaving the answer blank as an equal alternative. It's still very dubious as to whether there's any effectiveness, though. – Joe Z. May 28 '14 at 14:59
  • 4
    A possible problem with multiple choice tests is not necessarily about whether students can guess an answer, is't about whether the approach to learning has been to get students to answer this question correctly (teaching to the test) and sacrificing teaching for understanding along the way. It's conceivable (and shown in research) that students can get correct answers (legitimately) and lack understanding.

    The question "effective for what" still stands. Even if they do "measure what they measure" effectively, they may be measuring the wrong thing.

     – JPBurke May 28 '14 at 20:00
  • More importantly, they may be an indication of teaching the wrong thing. I stress may be. I don't think this is specifically an attribute of multiple choice questions, and it can certainly happen even if multiple choices are not used. – JPBurke May 28 '14 at 20:02
  • @JPBurke I'm trying to isolate multiple choice questions amongst other similar types of assessment. I think that your objections can be levied at any form of assessment and therefore are orthogonal to what I'm interested in. – Andrew Stacey May 28 '14 at 20:08
  • This is partly why my response is a comment and not an answer.

    But to help me understand the type of research you're looking for: if you were going to research your own question, can you give me an example of what sort of analysis you would apply? For instance, what would you do to see the effect you conjecture may exist, and how would you attribute it to guessing? I ask because I'm not clear on what sort of researchable question this is. That's partly why I tried to address my earlier comments to what is known about math education.

     – JPBurke May 28 '14 at 21:00
  • @JPBurke I can't really answer that as, not being an education specialist, I don't know the questions to ask. But I would start with two exams in the two styles that I suggest in the question and go from there. – Andrew Stacey May 28 '14 at 21:13
  • 2
    I admit that most of the problem here is with me. I'm a math education researcher and not a psychometrician. Although I think Matt F. asked a very relevant question. With two tests, all you can know is the difference in the test scores, not effects guessing might have introduced, assuming there is a known effect. But, really, I bet a psychometrician could tell you something a lot more useful than I could. – JPBurke May 28 '14 at 21:32
  • 3
    Probably students who are "exam-clever" do well at both multiple choice (any kind) or not multiple choice, and students who are not exam-clever do badly at both. So most assessments of exam-cleverness will give highly correlated results. But the type of exam given distorts what students take the trouble to learn, and often what is taught. – André Nicolas May 29 '14 at 00:48
  • @Andrew: Have you heard of the Force Concept Inventory? It seems like a pretty good example of a multiple choice test which tests for understanding. – Steven Gubkin May 29 '14 at 14:39
  • @StevenGubkin No, I haven't. Do you have a link to some details about it? – Andrew Stacey May 29 '14 at 18:39
  • https://modelinginstruction.org/wp-content/uploads/2012/08/FCI-TPT.pdf – Steven Gubkin May 29 '14 at 19:20

3 Answers3

5

I've read your question a few times, plus the comments. As best I can determine, your gut feeling is that constructed response exams should be a much better indicator of student understanding than multiple choice exams in math, because a) students cannot guess and b) partial learning can be recognized by analyzing the worked problem.

Therefore, you want to see a comparison of student performance on multiple choice versus short-answer questions testing the same concepts within the same exam. If the average student's performance is similar on the two parts of the exam, this would disprove your hypothesis that short-answer is better. Is this correct?

This question is a month old, but I'll assume you as OP think this does state your question and I will continue. I've listed some research articles below. The short answer is that when the question stems are matched, there is little difference in student strategy or student performance. It is possible that very carefully written CR exams might produce more information about student learning, but it is also likely that this careful question-writing hardly ever happens. My 20-min review indicates there is surprisingly little evidence that CR exams are better than multiple choice.

Lukhele, R., Thissen, D., & Wainer, H. (1994). On the Relative Value of Multiple‐Choice, Constructed Response, and Examinee‐Selected Items on Two Achievement Tests. Journal of Educational Measurement, 31(3), 234-250. They found student answers for AP Chem and AP History did not differ significantly between MC and constructed response.

Katz, I. R., Friedman, D. E., Bennett, R. E., & Berger, A. E. (1996). Differences in Strategies Used to Solve Stem-Equivalent Constructed-Response and Multiple-Choice SAT [R]-Mathematics Items. RESEARCH REPORT-EDUCATIONAL TESTING SERVICE PRINCETON RR. Found that matched-stem problems triggered similar student problem-solving and similar results.

Shepard, L. A. (2008). Commentary on the National Mathematics Advisory Panel recommendations on assessment. Educational Researcher, 37(9), 602-609. This is a short critique of research that matches MC vs CR question stems. They argue that CR allows for different sorts of testing and this should be taken into account.

Adrienne
  • 1,544
  • 10
  • 9
  • 1
    long-term, the problem with MC, especially if they become standardized and known to instructors, is that the instructors cease teaching the full spectrum of concepts and yield to the temptation to teach to the test. Do the studies above try to quantify this effect? – James S. Cook Jun 25 '14 at 14:12
  • Sorry - that's a completely different experimental test, and not part of these studies (or the OP's original question). I would argue (as someone who regularly administers exams to 800 undergraduates) that there are many ways to teach badly and we shouldn't throw out MC exams because some instructors teach to them. They can be written carefully and used effectively. – Adrienne Jun 25 '14 at 14:22
  • Thanks for the answer, and much to read through. However, your second paragraph gets it almost completely wrong: "If the average student's performance is similar on the two parts of the exam" The problem that I have with MC is that there is an unnecessary additional random factor introduced. On average, this will have no effect. But on individual students then it could potentially have a large effect. – Andrew Stacey Jun 25 '14 at 19:11
  • As best I can tell, these studies compare within-student results for both MC and CR, and find a single student produces similar scores in both types of exams. I'm unclear why you expect randomness to have an effect -- if an exam has 50 questions, wouldn't randomness have essentially equal positive and negative effects, and a student score be pretty close to their ability to solve math problems correctly? – Adrienne Jun 25 '14 at 20:11
  • I think Loop Space is worried about the fact that, for example, in the exam I just recently graded, there were ~10 students (out of 400) who completely, randomly guessed on a 30 question MC exam. From those students alone I have a swing of 15 points out of 100: one received 8 points more than complete random guessing would give on average, another received 7 points fewer. To some, a 15% swing is acceptable. To some, it is not. – Willie Wong Jun 30 '14 at 13:49
4

I don't know of research on the use of multiple choice questions for summative assessment specifically in mathematics. However, what's important is the approach to construction of multiple choice questions, if they are to overcome challenges like the ones you list. Following course design basics from sources like Wiggins & McTighe, what we know about assessments are that they are most effective when they are aligned with your learning goals.

With this in mind, I did come across one article that looks promising, but at the moment I only have access to the abstract, so I can't guarantee its value. (Edit: The article is available in full here.) From the abstract it appears the authors are taking a somewhat nuanced look at assessment in mathematics, and considering connections between the types of things you might want to assess, and how different assessment tools (in particular MCQs) can be of value.

Benjamin Dickman
  • 18,470
  • 2
  • 41
  • 114
Ruth
  • 41
  • 1
0

I prefer giving multiple choice exams. I can give frequent feedback to students, since the grading is easier. Below is a study the indicates no difference between multiple choice and short answer.

http://www.ncbi.nlm.nih.gov/pubmed/24274234

Chris

Chris
  • 1
  • 1
  • Thanks for the link. The abstract of the article makes it pretty clear that this is about formative assessment which I specifically rule out. Your answer also seems more formative in spirit since you talk about giving feedback to the students afterwards. – Andrew Stacey Aug 28 '14 at 19:17