2

It is well agreed (for example this discussion ) that Logistic Regression guarantees that model will produce well calibrated in the large (mean) predictions.

Does Logistic Regression guarantees that calibration-in-the-small is good too, or not necessarily?

If calibration-in-the-small is not guaranteed to be good (or in case when it is not good), what is known about the reasons?

Are there known methods to improve calibration-in-the-small via some modified loss function (or some other ways) while training LR model, or this is only solved by using some calibration function/transformation that is applied on outputs from prediction?

kjetil b halvorsen
  • 77,844
viggen
  • 41
  • 2
  • 1
    Could you direct us to the actual quotation? I can locate the comment about calibration-in-the-small and "calibration-in-the-tiny," but I am unable to find any comment that agrees with your initial statement. – whuber Jan 17 '20 at 21:13
  • Whuber, edited orginal question with details – viggen Jan 18 '20 at 05:42
  • You still seem to be reading something into Frank Harrell's comment that isn't there. The word "only" does not appear. – whuber Jan 18 '20 at 14:56
  • 1
    @whuber

    Frank wrote: "This addresses only calibration-in-the-large which is not what we want: calibration-in-the-small. – Frank Harrell Aug 20 '18 at 12:35"

    Let me ask different way : does logistic regression guarantee calibration-in-the-small? My experience is (on range of model types trained/fitted to very large amounts of data) - we get almost perfect calibration in the large, and ALWAYS severe lack of calibration in the small.

    So I am trying to understand if we have explanation why in-the-small happens to be not calibrated and what are common techniques to solve this

     – viggen Jan 18 '20 at 20:08
  • 2
    Prof. Harrell was referring to the answer under which his first comment appeared. That comment was not making a statement about logistic regression in general. I agree that you have a legitimate and interesting question here (which is why I did not vote to close it), but would prefer that it not be based on a possible mischaracterization of other work. – whuber Jan 18 '20 at 20:12
  • 1
    @Whuber thank you for clarifying - I corrected question to avoid mischaracterization (sorry for that) and genuinely interested in hearing advice about this problem. – viggen Jan 18 '20 at 23:10

0 Answers0