1

Consider a model of the probability of a binary, yes/no-type of event. The event is infrequent, say it happens only once every thousand times. In that regard, the prior probability of the event is $0.001$.

Consequently, if a predictive model like a logistic regression predicts a probability of $0.05$ given a certain situation (the model features), while there is still only a $5\%$ chance of the event happening, the event is $50$-times more likely to occur than usual.

$$ \dfrac{0.05}{0.001} $$

If that event is something catastrophic, I would want to know if the chance of it happening if $50$ times higher than usual, even if the event remains unlikely ($5\%$).

What drawbacks might there be to looking at predicted probability in this way? My reservation is that I don't want to get hung up on something like, "The chance of it happening is up from ultra-super-duper-unlikely to ultra-unlikely," something like a change in probability from a prior of $0.000001$ to $0.0001$. At the same time, a $100$-fold increase in event probability seems like a big deal, even if the event remains unlikely.

Dave
  • 62,186
  • Nice question (+1). Some quick points: 1. Such low probability scores are usually helpful to contextualise as part of their associated expected cost. 2. Visualisation of these results might be helped with the use of lift curves. 3. Associated with point 1. Can you look into the NetBenefit of the model? Some form of (clinical) usefulness? 4. Think a big draw-back is that quantifying "uncertainty" is pretty hard. Do we show a calibration plot focusing at the[0,10%] interval and show that we have well calibrated probabilities there for example? Tall order... – usεr11852 Mar 05 '23 at 00:18

1 Answers1

2

It depends on what you want to do with that prediction.

If you just want to know if there is information about some event in the predictors or compare models/predictors etc., then don't threshold and just analyze the raw predicted probabilities.

If you should just make a model, then make a model that outputs probabilities and let whoever uses the model use the probabilities to threshold them as they wish.

If you need to make a discrete action, then fit a model that predicts probabilities and then decide on the action based on a cost-benefit analysis. If a probability of an event is 0.2, it doesn't matter if the class frequency is 1/1 or 1/10000. What matters is only the cost of action/inaction and the cost of false positive/false negative predictions.

If you know that, as you said, "The chance of it happening is up from ultra-super-duper-unlikely to ultra-unlikely," you can make a lot of money if you can bet on it repeatedly, and a lot of people do, but in medicine, it probably doesn't matter

rep_ho
  • 7,589
  • 1
  • 27
  • 50