1

I am trying to get an intuitive understanding of the concept of calibration.

Definitions first. Consider a data distribution $P(X, Y)$ over binary labels $Y$, and a probabilistic classifier which returns the class prediction and the confidence estimate: $ h(X) = (\hat{Y}, \hat{P}) $. The confidence estimates are calibrated when:

$$ P(\hat{Y} = Y \mid \hat{P} = p) = p $$

What happens if $h$ is a (probabilistic) classifier which perfectly matches the true conditional $P(Y \mid X)$? Do we have perfect calibration then? Is it possible to have perfect calibration without $h$ perfectly matching $P(Y \mid X)$?

usual me
  • 1,227
  • 2
  • 10
  • 17

1 Answers1

1

Calibration reflects how well the predicted class probabilities match the 'true' probabilities according to the underlying distribution of the data.

as described here and here. If you are predicting the true probabilities, than they obviously perfectly agree with themselves, so it's perfectly callibrated.

Tim
  • 138,066
  • If calibration is about the difference between two distributions (true conditional vs model), why not simply consider the KL divergence between them? Why define this convoluted property above? Is it because it is easier to estimate in practice? – usual me Dec 21 '22 at 13:41
  • 1
    @usualme there are many ways of measuring calibration. Also, this is not something you can calculate directly: you don't know the true probabilities, so cannot measure it. If you knew the true probabilities, you wouldn't need to build a classifier... – Tim Dec 21 '22 at 13:51