3

I've the following situation.

I've a binary classifier which classifies input feature vectors into either of two classes '$y$' or '$n$', along with the probability of it being in either of the classes $P_y$ and $P_n$. I know that the classifier is right 60% of the time. How do I estimate the true probability of it being in each of the classes?

What I'm doing now is to compute $0.6*P_y / (0.6*P_y + 0.4*(1- P_y))$. This is a Bayesian adjustment, but I'm not sure if I'm doing the right thing.

kjetil b halvorsen
  • 77,844
broccoli
  • 1,048
  • Is this a homework problem?

    If not, how do you know that the classifier is right 60% of the time without knowing the true probability of each class?

     – Peter Flom Sep 22 '12 at 21:27
  • 1
    No, not a HW problem. I just left out that part of the detail. Basically, I create a classifier, using 50-50 y/n cases, followed by testing on unseen data. I note the number of times it the classification right. – broccoli Sep 23 '12 at 00:31
  • Sorry, but I don't understand your question. You know the prior probabilities for both classes (because you set them 50:50), right? You can count how often the classifier predicts each of the classes, right? These two are the marginal sums of the confusion table (the number of correct predictions is the trace). Also, I'm not completely clear what each of your "it"s refers to. Wrt. your Bayesian adjustment, you just give us a term, no equation. That means, that we cannot possibly tell you whether it is correct. – cbeleites unhappy with SX Sep 23 '12 at 15:44
  • 1
    Seeking a 'classifier' is at odds with estimating probability of class membership. Pick one or the other. If you like probabilities (which is a good idea), there is no such thing as being 'right' or 'wrong', there are just degrees of being right or wrong. – Frank Harrell Mar 30 '15 at 12:26

1 Answers1

0

If the model has calibrated probabilities, then this is exactly what the predicted probabilities tell you.

Calibration refers to the idea that, if a model predicts that an event will happen with probability $p$, then it should actually happen with probability $p$. If this happens, then the model outputs can be taken literally.

If the model lacks calibration, there are methods to try to calibrate the outputs so you can have this interpretation of the model outputs as the desired true probabilities of event occurrence.

All of this can be done independent of the accuracy score ($60\%$ for your problem), which is based on binning the probabilities in a way that might or might not make sense for a given application. For instance, despite having two categories, there might be three decisions you make, depending on the predicted probability. Our Stephan Kolassa gets into this idea here and here.

Dave
  • 62,186