0

I am currently analysing inter-rater reliability/agreement data for a single case with multiple raters. For that I am using Gwet's $AC_2$ (as described here) using the irrCAC package in R, written by Gwet himself. Unfortunately, I am not able to obtain a variance, SE etc. for my point estimate, as the code for this calculation is branched of in an if statement, that is only triggered when n>=2 (n referring to the subjects to be rated). Otherwise, the dispersion metrics are only reported as NA.

In my case of only a single subject, it would therefore not be possible, to calculate an error for my point estimate. I would assume that this if statement is necessary, because the variance calculation for Gwet's estimate contains a $1/n-1$ (Bessel's correction), which would lead to a division by 0 (when $n=1$). This can also be seen in formula (1) on page 413 in the linked paper:

Formula (1) from Gwet, 2008

This lead me to further, more general thoughts on the nature of variance:

1: a) Can the population variance in case of a single case just assumed to be 0? or

b) Is it generally not possible to calculate a population variance with the Bessel correction of $1/n-1$ for a single case sample, i.e. is the NA in Gwet's code the correct representation (because it just cannot be calculated)?

2: Is there a special way of calculating the variance for a single case situation?

I was reading this question on implication for the mean in case of n=1 and tried to apply its concepts for my case, but did not manage to find a conclusive answer.

Dom42
  • 85
  • 6
  • You can, if you want, redefine the sample variance to be divided by $n$ instead of $n-1$. However, when you only have 1 data point how do you even measure spread when there is no other competing quantity? A more serious question is, why would you use data that is just 1 number? Any conclusions drawn from that will be very suspicious. – Nicolas Bourbaki Feb 12 '23 at 23:15
  • Thank you for your input! I also had the idea of just using $n$ but I was unsure if that is a valid approach. As I mentioned, I am also unsure if measuring spread is "allowed" in such a case (therefore question 1b). In case of inter-rater reliability studies, such a case can arise if there is really only one subject that the raters can judge (e.g. because there is only one case in a clinic to be judged and all other cases are somewhere else). Therefore the question is born "out of necessity" rather than what is wished for. – Dom42 Feb 12 '23 at 23:34
  • There is no right way to handle just 1 data point. It would be a type of research that other people might say is lacking data to draw conclusions from. Perhaps, it is possible to group those single entry numbers into a common group which is believed to be similar? If that is not possible to do such a grouping, then it would be better to see if you can somehow collect more data. – Nicolas Bourbaki Feb 12 '23 at 23:40
  • 1
    Perhaps, you might find this question relevant? https://stats.stackexchange.com/questions/157582/what-can-we-say-about-population-mean-from-a-sample-size-of-1?noredirect=1&lq=1 – Nicolas Bourbaki Feb 12 '23 at 23:42
  • I guess, the $n$ workaround would satisfy my needs (with a big disclaimer that this is based on a single case). Unfortunately, currently only such a single case is available and I need to find a solution. Concerning your link: this is the question I also linked in my post. Unfortunately, I was not able to draw good conlcusions from it. Nevertheless, thank you! – Dom42 Feb 12 '23 at 23:49

0 Answers0