Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [cohens-kappa]

A measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

Cohen's kappa is a measurement of the degree to which 2 rating systems (typically people making ratings) agree. It adjusts for the probability that the raters would agree by chance alone if they were completely independent.

The sampling distribution of kappa is known, so the measure can be used as a statistical test of agreement.

The standard calculation of kappa is: $$ \kappa=\frac{p(\text{agreement})-p(\text{chance agreement})}{1-p(\text{chance agreement})} $$ There are also weighted versions of kappa, and other related measures of inter-rater agreement.

193 questions
6
votes
1 answer

Kappa Statistic for Variable Number of Raters

I have $N$ documents. Each document $N_i$ is annotated by a variable number of raters $n_i \in [3, 10]$ into two categories. Is there a way to compute the Kappa statistic for these annotations? My understanding from Wikipedia is that Fleiss' Kappa…
mossaab
  • 229
4
votes
2 answers

Quadratic weighted kappa strength of agreement

In the case of the kappa-value there are some attempts to qualify how good or bad the agreements are. For example Landis & Koch in the article The Measurement of Observer Agreement for Categorical Data talks about "strength of agreement" based on…
andreSmol
  • 537
4
votes
2 answers

How to calculate maximum value of kappa?

Cohens kappa can be used as a measure of interrater agreement. However, sometimes the theoretical maximum of kappa < 1 and it may be more correct to calculate kappa as the proportion of the maximum value of kappa. I need a good calculation example…
Kasper Christensen
  • 620
3
votes
0 answers

Is there such a thing as a multidimensional kappa test?

I am involved with designing a new diagnosis method for a routine clinical procedure. I want to compare the agreement between operators to the agreement between methods. Is there such a thing as a multidimensional Kappa test, and if so how do I…
Doragan
  • 69
  • 3
2
votes
1 answer

Cohen's kappa with three categories of variable

I have to calculate the inter-agreement rate using cohen's kappa. However, I only know how to do it with two observers and two categories of my variable. My task now is this: I have a set of tweets. Each tweet should be rated as…
Assen
  • 23
1
vote
0 answers

Analysing agreement with many dimensions to the data

I am tasked with running a sample size calculation and also statistical analysis of a study looking at the agreement between two 'tools'. This is a binary outcome so looking mainly at the different forms of kappa. However, here is the complexity to…
Dean Langan
  • 11
1
vote
0 answers

How to compare two Kappa values based on the same sample but different methods (two new methods A and B, and a gold standard method C)

Now, I have two assessment methods A and B (both of them are new methods) for assessing diabetes. Moreover, the method C is the gold standard. Firstly, I ask a rater (Raters X) to use Method A and C (gold standard) to assess 50 individuals,…
skywalker21th
  • 59
1
vote
2 answers

Strange values of Cohen's kappa

I have two raters who agree on 93% on the cases (two options: yes or no). However, when calculating cohens kappa through crosstabs in spss I get really strange outcomes like -0.42 with a sig. of 0.677. How can such a high agreement in percentage…
Karien Lomans
  • 11
1
vote
1 answer

weighted cohens kappa categories

I need to ask about how many categories can be included in a weighted kappa or cohens kappa statistic for inter observer reliability. Effectively we need to be reliable in assessing whether the distance between two monkeys is: 0m, 1m, 3m, 5m or…
sarah
  • 13
1
vote
2 answers

Explanation of the denominator in Cohen's kappa score?

Cohen's kappa with two raters is: $$\kappa = \frac{p_o - p_a}{1 - p_a}$$ where $p_a$ is probability of agreement by chance, and $p_o$ is the observed rate of agreement. I can't figure out why the denominator is $1 - p_a$ instead of $p_a$. Shouldn't…
Chip Huyen
  • 111
  • 2
1
vote
1 answer

Linear and Quadratic Weighted Kappa SPSS Error

I am using both Linear and Quadratic weighted kappa to examine the extent to which two measures (a quantitative clinical interview, and a corresponding self-report questionnaire) agree at the item level. By this I mean that I am checking extent to…
Alice Phillips
  • 11
1
vote
1 answer

Explain Cohen's kappa in a simplest way?

I'm learning statistics but don't really understand the result of kappa value posted in the first sample here. Is there anyone help me explain in English what the value 0.915 means? (observed proportionate agreement) and the kappa value 0.801. Does…
Peter Sm
  • 11
1
vote
2 answers

Cohen's Kappa, why not simple ratio

I was wondering if there is any particular reason why Cohen's Kappa is defined as this particular ration $\frac{p - r}{1 - r}$, where $p$ is the agreement rate between two, say, classifiers, and $r$ is the rate of agreeing at random. Why is Kappa…
inzl
  • 1,283
1
vote
2 answers

How can this rating have negative Fleiss Kappa?

I am trying to calculate the Fleiss' Kappa of the following data and I got the result: [[0 0 3] [0 1 2] [0 0 3] [0 0 3] [0 0 3] [0 0 3] [0 0 3] [0 0 3] [0 0 3] [0 1 2] [0 0 3] [0 0 3] [0 0 3] [0 0 3] [0 0 3] [0 0 3] [0 0 3] [0 1…
Zhiya
  • 241
0
votes
1 answer

Cohen's kappa for agreements between two inter rater metrics?

I have two metrics for measuring source code cohesion. To verify that both have some agreement can I use Cohen kappa for that? Both the metrics have values in range [0 to 1]. It's not necessary if both have the same values. If both the metrics are…
alo
  • 1
1
2