1

I have some results where the tester claims the following values:
Sensitivity: 0.525, Specificity: 0.925, Precision: 0.516, Accuracy: 0.907

Where
Sensitivity=TP/(TP+FN),
Specificity=TN/(TN+FP),
Precision=TP/(TP+FP),
Accuracy=(TP+TN)/(TP+TN+FP+FN)

I'm having trouble seeing how these reported values are possible; are they?

James Brown
  • 23
  • Looks like TN vs FP is high, but TP vs FP,FN is low. Your problem lies in TP. It means your classifier is mostly right when a sample test is negative, but unreliable in the opposite case. This is plausible (though not satisfying). – jpmuc Jul 28 '14 at 12:16
  • The reported values do not fully fit together. It is possible to generate a 2x2 table, where the reported sensitivity, specificity and precision hold, but the reported accuracy can't then be right. For example, assuming the following numbers of observation: TP=160, FN=145, FP=150, and TN=1850 gives the right values for other measures, but not accuracy. –  Jul 28 '14 at 12:21
  • Thanks for the comments. JTT, as you've stated, the maths doesn't add up; accuracy will work out at 0.872 with all others matching exactly. – James Brown Jul 29 '14 at 02:40

1 Answers1

4

Let $\text{TP}=a, \text{FP}=b, \text{TN}=c, \text{FN}=d$.

The given information is:

$a = 0.525\ (a+d)\\ b = 0.925\ (b+c)\\ a = 0.516\ (a+b)\\ (a+c) = 0.907\ (a+b+c+d)$

So

$d = 0.475/0.525\ a = 0.90476\ a\\ b = 0.484/0.516\ a = 0.93798\ a\\ c = 0.075/0.925\ b = (0.075/0.925)\ \times\ (0.484/0.516)\ a = 0.07605\ a$

and substituting into $(a+c) = 0.907\ (a+b+c+d)$ we get:

$(1+0.07605)\ a = 0.907\times (1+0.93798+0.07605+0.90476)\ a$, but

$(1+.07605) \neq 0.907\times (1+0.93798+0.07605+0.90476)$.

So yes, it's inconsistent - no set of TN, TP, FN, FP can satisfy those.

Looking at sensitivity and precision, we have $d = 0.475/0.525\ a = 0.90476\ a$ and $b = 0.484/0.516\ a = 0.93798\ a$. They together imply that accuracy lies between 0.351 and 1, so they're not inconsistent with that given accuracy.

Looking at specificity and precision, they together imply that $(a+c)/(a+b+c)=0.5343$, meaning that accuracy must be less than 0.5343.

Since sensitivity and precision were consistent with accuracy, and with specificity, but those two don't seem to go together, it looks like one of specificity or accuracy would be the likely culprit.

Glen_b
  • 282,281