1

In his Is Medicine Mesmerized by Machine Learning? blog article, Frank Harrell shows a calibration curve (below) and states that it is quite poor.

calibration curve

I follow the logic: the claimed probability of $0.20$ corresponds to an actual probability of $0.08$-$0.10$. Doubling the true probability sure seems like poor performance.

What are the good ways to wrap this up into one number? My first thought is to integrate to find the areas above and below the "perfectly calibrated" line. Are there others that work better or give different insights?

Dave
  • 62,186
  • We should keep in mind that the curve (as well as some sort of calibration summary) is produced from random data. So although it looks poor, the model which produced it might be fine. – picky_porpoise Nov 10 '23 at 20:04
  • @picky_porpoise Fair point, but given the fairly large deviation from the ideal and the fairly large sample size, some kind of confidence band on the plot would likely be quite narrow and away from the 45-degree ideal. – Dave Nov 10 '23 at 20:08

0 Answers0