1

I am working on a dataset that has three raters. Rating are Yes/No. I have a set of ratings where all raters said No. I used a R package and excel formulas to calculate the kappa score. Both return NaN. I get why it returns NaN. Because the expected probability is 1, the denominator is 0. Hence the NaN. It seems the case is valid when all raters mention Yes to all instances.

Isn't this a perfect agreement?

Here is the data to reproduce

#    Rater 1    Rater 2    Rater 3    Yes    No
1        Yes        Yes        Yes      3     0
2        Yes        Yes        Yes      3     0
3        Yes        Yes        Yes      3     0
4        Yes        Yes        Yes      3     0
5        Yes        Yes        Yes      3     0
6        Yes        Yes        Yes      3     0

Pe = (sum(Yes)/Nn)^2 + (sum(No)/Nn)^2
   = 18/63 + 0/63
   = 1


Po = 1/(Nn(n-10)(Sum_of_squares(all_ratings) - Nn)
   = 1/(632) * ((3^2+3^2+...+0^2+0^2)-63)
   = 1/36  (54-18)
   = 1/36 * 36
   = 1


kappa = Po - Pe / 1 - Pe
      = 1 - 1 / 1 - 1
      = NaN
akalanka
  • 111

1 Answers1

0

Your by-chance error rate is zero, since all of the raters give the same “yes” every time. Your kappa calculation, in some sense, doesn’t even know that “no” is an option for the raters. When there is just one option, it doesn’t really make sense to calculate an agreement statistic: perfect agreement is inevitable.

You might be interested in something along the lines of Laplace’s rule of succession to handle the fact that there are only “yes” responses in the data, yet you know “no” to be an option.

Note that this is not an issue if the raters give both responses yet agree every time, something like each rater saying YNYYNN

Dave
  • 62,186