0

If we have a model which outputs class probabilities for $K$ classes, e.g. a NN with softmax layer, how can we return an aggregate "confidence metric"?

Some intuitive ideas would be the probability of the highest class or the difference in probability between the top 2 classes.

Is there a standard choice here? If not, what are the options.

curiousgeorge
  • 349
  • 1
  • 10
  • What does the value of the loss function (probably categorical crossentropy) not give you? – Dave Oct 30 '20 at 03:00
  • Well the cross entropy in most contexts would just be $-log(p_k)$ where $p_k$ is the probability class $k$ with highest probability. So this only takes into account the max value. However, we may want to consider the difference between top 2 values for example which is useful. – curiousgeorge Oct 30 '20 at 03:06
  • Perhaps you can say more about what a confidence score is. Crossentropy loss is a strictly proper scoring rule that aims to find the correct probabilities, so if you want to measure if your model is close to modeling the phenomenon under consideration, that’s a pretty good measure. – Dave Oct 30 '20 at 03:18

0 Answers0