1

I have ordinal predictions by experts: 1, 2, 3, 4, 5. The outcome to be predicted is, however, continuous: 0-36.

I don't know what threshold the experts use for the categories 1-5, but I know that the outcomes in 1 should be smaller than in 2, in 2 smaller than in 3, and so forth. How can I compute the accuracy of each expert?

Intuitively, I would like to allow for different thresholds for each expert. So I would basically count the ranking errors.

Are there any established methods to do this?

Stephan Kolassa
  • 123,354
jkortner
  • 111
  • I took the liberty of removing the [tag:scoring-rules] tag, because (proper) scoring rules have a very precise technical meaning, being evaluation measures for probabilistic predictions, e.g., here for ordinal data - and you don't have that. That said, you might be able to use a kind of rank correlation, like Spearman's or Kendall's? I know it's late, but can you give any info on what exactly your experts rate, what the outcomes are, and how many realizations per expert etc. there are? – Stephan Kolassa Jul 07 '23 at 13:06

0 Answers0