1

I have two raters who agree on 93% on the cases (two options: yes or no). However, when calculating cohens kappa through crosstabs in spss I get really strange outcomes like -0.42 with a sig. of 0.677.

How can such a high agreement in percentage result in such a strange kappa?

I don´t get it.

Karien Lomans
  • 11

2 Answers2

2

Suppose there are 100 ratings. In 98 of them, both raters say "Yes". In 1, rater A says "no" and rater B says "yes"; in 1, just the opposite happens. Now kappa is near 0 (actually slightly below 0) but agreement is 98%. If you increase the '1's you lower both kappa and agreement.

Kappa is, in a way, designed to do this. It is intended to reduce the influence of matching that isn't indicative of anything.

Peter Flom
  • 119,535
  • 36
  • 175
  • 383
  • With 93% agreement, I am having a very hard time achieving a value of kappa any less than $-3/77 \approx -0.039$. – whuber Jan 17 '13 at 21:08
  • @whuber Well, that's why I asked the Karen to post her crosstabs, but she hasn't done so. – Peter Flom Jan 18 '13 at 12:02
  • If I'm correct, that means your answer cannot possibly explain what the O.P. is reporting, no matter what her data may be. – whuber Jan 18 '13 at 14:03
1

A concept is not well defined (learnable) if you only have positive examples of it - it could be anything between the specific set of positive examples and the whole universe.

The problem, as illustrated with Peter's example comes where there are no (or very few) examples where the raters agree on "no"! Presumably this is also what Karien and whuber have. It would of course be just as bad to have no (or few) where the raters agree on "yes"!

With the symmetric agreement/disagreement pattern each way, the Kappas of Cohen and Fleiss and Scott all agree, as do DeltaP, DeltaP', Informedness, Markedness and (Matthews) Correlation - which are focussed on chance-correct prediction of one set of ratings using the other, and give the probability that such a prediction is informed.

Let's suppose they agree yes on A, disagree one way on B, the other on C, and agree no on D:
A | B
- - -
C | D

The symmetry gives us B = C and the lack of negative agreement gives us D = 0.

Exploiting symmetry and using the simpler (and I believe sounder) Informedness or DeltaP formulation we have Kappa = A/[A+C] - B/[B+D] = A/[A+C] - 1.

This will be <0 (below chance) except when B=C=0 - both raters agree all examples are yes, when Kappa = 0 indicating chance level performance (there is no work for them as the examples appear to be all positive and all uncontroversial).

Without the symmetry the Kappas and Correlations are still negative although DeltaP and DeltaP' (calculated on the transposed table) differ and so the magnitude of the Correlation is their geometric mean (Cohen and Fleiss Kappa can give weird results for extreme asymmetry too - see the reference below).

Reference

Powers, D.M.W (2012). The Problem of Kappa. In Conference of the European Chapter of the Association for Computational Linguistics. ACL.
http://aclweb.org/anthology-new/E/E12/E12-1035.pdf

David M W Powers
  • 1,441
  • 14
  • 12