Given a confusion matrix of a binary classifier, what are the best measures of response bias towards one of the classes?
One idea that comes to mind is Signal Detection Theory's criterion, but this measure assumes a Gaussian noise model.
Is there a more straightforward, well-tested measure of response bias? For example, we could divide the number of predicted positives (TP + FP) by the number of positives (TP + FN), but I'm not sure that such a ratio would be "well behaved".
The Python package sdt_metrics by Roger Lew implements several non-parametric response bias measures. Unfortunately, the package is not maintained, but the references are still useful.
One of these references is an empirical study comparing five response bias measures:
See, J. E., Warm, J. S., Dember, W. N., and Howe, S. R. (1997). Vigilance and signal detection theory: An empirical evaluation of five measures of response bias. https://doi.org/10.1518/001872097778940704
Among the parametric response bias measures, they recommend B"D: $$B’’_D = \frac{(1-H)(1-FA)-(H)(FA)}{(1-H)(1-FA)+(H)(FA)}$$
Note that care must be taken when $H$ (hit rate) or $FA$ (false alarm rate) are at their boundaries:
Hautus, M.J. Corrections for extreme proportions and their biasing effects on estimated values of d′. Behavior Research Methods, Instruments, & Computers 27, 46–51 (1995). https://doi.org/10.3758/BF03203619
I'll expand on my comment here. This answer was partially adapted from this one.
Let $Y$ represent a class label and $\hat Y$ represent a class prediction. In the binary case, let the two possible values for $Y$ and $\hat Y$ be $0$ and $1$, which represent the classes. Next, suppose that the confusion matrix for $Y$ and $\hat Y$ is:
| $\hat Y = 0$ | $\hat Y = 1$ | |
|---|---|---|
| $Y = 0$ | 10 | 20 |
| $Y = 1$ | 30 | 40 |
With hindsight, let us normalize the rows and columns of this confusion matrix, such that the sum of all elements of the confusion matrix is $1$. Currently, the sum of all elements of the confusion matrix is $10 + 20 + 30 + 40 = 100$, which is our normalization factor. After dividing the elements of the confusion matrix by the normalization factor, we get the following normalized confusion matrix:
| $\hat Y = 0$ | $\hat Y = 1$ | |
|---|---|---|
| $Y = 0$ | $\frac{1}{10}$ | $\frac{2}{10}$ |
| $Y = 1$ | $\frac{3}{10}$ | $\frac{4}{10}$ |
With this formulation of the confusion matrix, we can interpret it as an estimate of the joint probability mass function (PMF) of $Y$ and $\hat Y$. This interpretation allows us to measure how often a certain class is predicted. For example, suppose we wanted to compute $P(\hat{Y} = 1)$, which is how often the classifier predicts class $1$. Using the law of total probability, we can compute this as $P(\hat{Y} = 1) = P(\hat{Y} = 1,Y = 1) + P(\hat{Y} = 1,Y = 0) = \frac{4}{10} + \frac{2}{10} = \frac{6}{10}$. Therefore, the classifier predicts class $1$ about $60\%$ of the time. Note, however, that these are all estimates.
