Deriving confidence from multi-class prediction probabilities

Question

I have a multi-class classification model that predicts probabilities of 4 classes with the following percent of True Positives per class 0.74, 0.86, 0.87, 0.91.

I have predictions for 3 test cases:

1. (0.6, 0.1, 0.1, 0.2)  
2. (0.01, 0.09, 0.5, 0.4) 
3. (0.55, 0.25, 0.1, 0.1)

Questions:

Would it be correct to say that class with probability 0.6 in case 1 is a confident prediction and class with probability 0.01 in case 2 is not a confident prediction?
And what can we say about class with probability 0.55, is it a confident prediction or not?
Would it be a better way to calculate 95% confidence interval for accuracy then trying to evaluate confidence of prediction probabilities?

Thanks!

Sorry for wrong description. 0.74, 0.86, 0.87, 0.91 are not probabilities but percent of TP in every class. — dokondr, Feb 06 '23 at 00:56
You don't add percents of different classes. You add percent of TP with percent of FP to get 100 % of samples in one class. This numbers are from confusion matrix. — dokondr, Feb 06 '23 at 01:06

Dave · Answer 1 · 2023-02-06T01:43:16.313

1

$1\space\&\space 2)$ A predicted probability of $0.01$ is an emphatic prediction. "I really do not expect this to happen," the model is telling you when it outputs such a prediction. On the other hand, a predicted probability of $0.6$ represents the model being fairly wishy-washy. "Maybe it will happen, but I am not particularly sure," the model is telling you.

$3)$ Accuracy has major problems (same for precision, recall, and $F_1$), one of which is that your model that outputs probabilities gets every single prediction wrong (or does it have any predicted probabilities of exactly $0$ or $1?$) and thus has $0\%$ accuracy. Calculating accuracy requires a threshold to be applied, and the linked post and links contained within explain why that is of questionable utility (and further discussion of this topic warrants a separate question, not a discussion in the comments). You might consider interval estimates for your probabilities, sure.

edited Feb 06 '23 at 01:43

answered Feb 06 '23 at 01:21

Dave

62,186

Please clarify this: "Consider what the predicted probabilities would be if you flipped the categorical encoding (zero to one, one to zero). Your old predicted P(Y=1)=0.01 would then be P(Y=1)=0.99 , while your old P(Y=1)=0.60 and P(Y=1)=0.55 would then be P(Y=1)=0.40 and P(Y=1)=0.45 , respectively. What do you mean by ' flipped the categorical encoding ' ? And also, I have 4 classes, not just 2. thanks – dokondr Feb 06 '23 at 01:26
@dokondr I have removed that discussion, since it does not apply (at least not in an obvious way) to a situation with four categories. – Dave Feb 06 '23 at 01:40
I see what you mean, thanks. On another point: Confidence interval may be calculated not only for accuracy, but for other statistics as well: P, R and F1, it should be more informative, I think. – dokondr Feb 06 '23 at 01:40
1

@dokondr This discussion warrants its own question (and there are plenty), but all of those statistics suffer from problems similar to that of accuracy. Please refer to the linked question and the links within. – Dave Feb 06 '23 at 01:42
Thanks, I will. It still will be great if you could give a short answer how to estimate confidence of class prediction in multi-class model. This short answer would work as a course for me to further investigate on my own, without asking more questions here. This was a whole point of my initial question here. – dokondr Feb 06 '23 at 01:51
That warrants a separate posted question that specifies what you mean by "confidence", as the term can have a technical meaning in statistics that is not the same as its meaning in colloquial English. – Dave Feb 06 '23 at 01:54

Deriving confidence from multi-class prediction probabilities

1 Answers1