3

I'm currently working on developing a language learning web app that provides the user with better feedback.

One of the things that I'm looking into right now is evaluating wrong answers. I'd like to implement a scoring scale for each question that the user answers by typing in the word.

A correct answer is +1.00, wrong / blank answer is 0.00, and typing in a different word would be -1.00 as it indicates the user may have learned the word wrong.

My question is, what should I look at to decide if a user is close? For a misspelling, I can use the Levenshtein distance which is basically the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other.

The problem is, forgetting an accent or perhaps using a wrong accent doesn't mean that the user doesn't know the word. Also, the length of the word should be considered as missing one letter in an eight letter word isn't a severe as in a three letter word.

What are your recommendations as to a scoring system between 0 and +1.00. How should I decide how close they are to the right idea and award partial credit accordingly?

Tsundoku
  • 19,740
  • 4
  • 43
  • 120
Seth Killian
  • 31
  • 1

1 Answers1

1

Write (or find a database of) incorrect example answers; at least ten of them. For each of them, consider how close they are to perfect. Use, for example, percent scores or points out of ten or whatever is intuitive for you. (A fairly good answer might be 8/10 or 80 %, for example.)

Next, try different scoring algorithms on your examples. See which best fits the scores you want to have. You can do this on an ad hoc basis, or if you know statistics or mathematics, you can create families of algorithms with a parameter and then optimize for the best value the parameter.

Finally, you need to convert your scoring algorithm from the scale 0 - 1 (expressed as 0 % - 100 % or 0/10 - 10/10, maybe) to scale -1 - 1. To do this to score x, calculate 2*(x-0.5).

I suggest working with the range from 0 to 1 at first, as it is more intuitive and easier to work with.

An example algorithm for scoring would be to consider the relative Levehstein distance; that is, the distance divided by the length of the word. For no errors, this gives value zero. If all the letters are wrong, this gives the value 1. It may give even larger values, but you can cap it at 1. This measures error. To convert it to a score, calculate 1-error. Then you can convert it to the interval between -1 and 1 as given above.

(Warning: Relative Levehstein distance might not be a standard term, as I came up with it without checking any sources.)

Tommi
  • 3,492
  • 1
  • 11
  • 37