Is it possible to adjust a logistic regression model when the false positive rate is too high for observations belonging to a specific category?

Question

This is a hypothetical scenario for self-learning purposes (not homework), so not a lot of additional details to give than what I mention below.

Let's say I have a binary logistic regression model, using various independent variables. Let's say that among these variables, there's a variable "profession".

I use the model for prediction: I classify observations in a category "likely to commit tax fraud" when the model prediction for an observation is above a certain level (say 0.65).

My decision rule produces a certain overall rate of false positives (say 0.1). However, I notice that the false positive rate is much higher (0.3) for people belonging to a certain profession.

This happens to be a problem, and I want to correct it, that is reducing the false positive rate for this specific profession, without affecting the false positive rate for other professions.

My questions are:

• Is the situation I describe even possible? (i.e. a false positive rate higher for some categories than for others).
• If it is possible, in general what are the reasons why it happens? Is there a way to correct the issue directly in the model? (e.g. by using weights, completely re-specifying the model, etc.). Or is the only solution to adjust the decision rule on a "per independent variable" basis, after the model predicted the value for an observation?

Answer 1

If your classification model is based on a certain output variable surpassing a specific boundary, then you can adjust that output variable with a certain amount such that it changes the amount of positives (and along with it the false positive rate). if you do this only for the specific category then it will only change the frequency of positives for that particular category.

These differences in false positives occur if

• a particular feature associated with tax fraud occurs a lot in the class with the high false positive rate.

  An example is discrimination based on racial features. If certain racial features correlate with tax fraud then these make a very good classifier, but it will create more false positives among people with those racial features. Also if racial features are not used in the classifier, they might correlate with other features that are used.

• the class has a lot of variation in the features or if there is little data for the class available.

  A similar effect is imbalance as described here: Was Amazon's AI tool, more than human recruiters, biased against women? . If people of a certain profession are not well represented in the dataset, then then the model might not predict the probability of fraud very well for those people. If you correct for this with a different boundary, then it comes at a cost with a different false negative rate. The bias remains and is not removed by changing the boundary. To get rid of that you need to improve the model for the particular class (e.g. more or better data, or better analysis).

---

Related note: The probability parameter that the logistic model uses is not the same as the false positive rate or true positive rate. It relates to the odds of the classes, but depending on the distance, variance and ratio of positive/negative a same odds-value-cutoff can give different false positive and false negative rates. See also the question: Probability threshold in ROC curve analyses

The image below explains this further. If we classify positive/negative based on some odds rate, e.g. if the odds are above 0.65 then we classify it as positive, then depending on the distribution of the positive and negative classes, you get different false positive and true positive rates.

It is also interesting to look at the following image from the previous referenced question

The image displays how a logistic regression can be used to classify two classes (0,1 in this case) based on two parameters/features (petal width and petal length in this case, in your case one of them could be 'profession' and it could also be a categorical variable).

Say that we select as classification cutoff: the line where $p=0.5$ (ie. equal probability of the two classes). In that case, then we will be making different false positive and true positive rates conditional on a single feature. For a petal length around 6 nearly all the flowers will be (correctly) classified as class 1. For a petal length around 5 there will be some false positives and the true positive rate is less high.