What do I do when a false negative is far more expensive than a false positive?

Question

I'm not sure how to 'responsibly' balance my model to account for this. I could predict a probability and give that to the business ('predict_proba' in SKlearn) but experience in the past has thought me that I should be the one (they put it on 0.9 because they felt more safe that way).

I'm considering resampling the training data in some way... Let's say I'm in charge of an algorithm that removes rotten apples, which is extremely hard (let's say that without bootstrapping it classifies 10% of the rotten apples incorrectly as good apples, and 20% of the good apples as bad apples, but not relevant for this question I think).

The ratio between good & bad apples is 1000/1.

The company sells apples for 50 cnts. Selling a rotten apple costs them on average 30 dollars as a customer will often return it and not come back. Catching a rotten apple also costs them 1 cnt for returning it too the farmer. Sending a good apple to the farmer costs them 4 euro's as it kills the relationship with the farmer and the apple gets returned.

Without any additional work on the features and modelling, could I resample (bootstrap) training data (ratio good & bad is 1000/1) in such a way to align with the cost ratio of the apple value, such that a model is most likely to create the most value? Or are there other ways?

Change your loss function so that false negatives are more expensive? — wzbillings, May 12 '23 at 12:59
@wzbillings Thanks & yep that sounds like a thing that could work! If you have experience with that please feel free to share your code/references :-) — Cdl, May 12 '23 at 14:19

score 11 · Accepted Answer · answered May 12 '23 at 13:23

11

Let's assume that your model is well-calibrated (see calibration). If it isn't you can calibrate it, if there are other issues they need to be solved accordingly. This does not need re-sampling of the data.

Then, the problem is picking the right threshold for making the predictions. You already seem to have all the pieces for doing it! The costs matrix for your problem is:

	is good	is rotten
predicted as good	50	-30
predicted as rotten	-4	-1

With this information, you can pick a threshold and make the positive prediction when the predicted probability is greater than the threshold. After doing this, for each prediction assign the appropriate cost from the matrix above, and calculate the average cost. You can do this for different thresholds and just pick the threshold that maximizes the expected cost.

answered May 12 '23 at 13:23

Tim

138,066

3

Related: Reduce Classification Probability Threshold – Stephan Kolassa May 12 '23 at 13:46
3

This answer shows a formula for the cost/benefit-based probability cutoff choice. – EdM May 12 '23 at 14:20
One question though: you talk about a threshold, would that be the 'predict_proba' of a SKlearn package or did you have a more sophisticated parameter in mind? – Cdl May 12 '23 at 15:25
3

@Cdl I mean thresholding the predicted probabilities (i.e. predict_proba). – Tim May 12 '23 at 15:52

What do I do when a false negative is far more expensive than a false positive?

1 Answers1