Is a pair of threshold-specific points on two ROC curves sufficient to rank classifiers by expected loss?

Question

We have two models outputting estimates of class probabilities. Combined with a probability cutoff / threshold, these yield classification decisions: if the estimated probability of class 1 is above the threshold, the assigned label is class 1; otherwise, it is class 0. We want to compare the models in terms of their estimated expected loss for a given threshold. The loss function $L(\hat{Y},Y)$ is given by \begin{aligned} L(0,0)&=0, \\ L(0,1)&=a, \\ L(1,0)&=b, \\ L(1,1)&=0 \end{aligned} with $a,b>0$. The estimated losses on a test subsample are not available. However, we have the ROC curves on the test subsample for each model. From each curve, we can obtain the point that corresponds to the optimal threshold as derived from the loss function. (By optimal threshold I mean the one that minimizes the estimated expected loss. For the general loss function specified above, it is $\frac{b}{a+b}$, assuming the estimated probabilities are reasonably well calibrated.)

Question: Does this pair of points on the ROC curves contain sufficient information to conclude which classification algorithm has the lower estimated expected loss? If not, could you offer a counterexample?

_{Related question: "Are non-crossing ROC curves sufficient to rank classifiers by expected loss?".}

Answer 1

I think the answer is no as the expected loss depends on $P(Y = 1)$, and this information isn't given by a ROC curve.

Let's say you have a binary random variable $Y$ with $p = P(Y = 1)$, and note $\hat{Y}_t$ a classifier depending on a treshold (or more generally a parameter) $t$.

The expected loss of classifier $\hat{Y}_t$ is $$ \begin{array}{ccl} L(\hat{Y}_t) &= &a P(\hat{Y}_t = 0 \cap Y = 1) + b P(\hat{Y}_t = 1 \cap Y = 0)\\ & = & a \cdot p\cdot P(\hat{Y}_t = 0 \mid Y = 1) + b \cdot (1-p)\cdot P(\hat{Y}_t = 1 \mid Y = 0). \end{array}$$

The ROC curve only gives you the conditional probabilities $P(\hat{Y}_t = 1 \mid Y = 0)$ and $P(\hat{Y}_t = 0 \mid Y = 1)$ as a function of $t$, but they don't give you $p$.

Consider for example two (very bad) classifiers $\hat{Y}_t$ and $\hat{Z}_t$ with $$ \begin{array}{ccl|ccl} P(\hat{Y}_t = 1 \mid Y = 0) &=& 1/2 & P(\hat{Z}_t = 1 \mid Y = 0) & = & t\\ P(\hat{Y}_t = 0 \mid Y = 1) &=& 1 - t & P(\hat{Z}_t = 0 \mid Y = 1) & = & 1/2 \end{array} $$ giving the following ROC curves.

The the expected loss of $\hat{Y}_t$ is $$L(\hat{Y}_t) = \frac{1}{2}(1-p)b + (1-t)a p$$ and the expected loss of $\hat{Z}_t$ is $$L(\hat{Z}_t) = t b (1-p) + \frac{1}{2} a p.$$ There is no way to tell which is the best without knowing $p$...

However, if a ROC curve dominates another, meaning it's above it all the time, then you known that whatever probability $p$ and whatever losses $a$ and $b$, the dominating classifier will have lower expected loss than the other (it follows directly from the expression of the expected loss).

Indeed, if the roc curve of $\hat{Y}_t$ is above the ROC curve of $\hat{Z}_t$, then for each $t$, the ROC curve of $\hat{Y}_t$ is either left or above (or both) of the ROC curve of $\hat{Z}_t$, this implies that $$P(\hat{Y}_t = 1 \mid Y = 0) \leq P(\hat{Z}_t = 1 \mid Z = 0)$$ and $$P(\hat{Y}_t = 1 \mid Y = 1) \geq P(\hat{Z}_t = 1 \mid Z = 1), $$ and thus $$P(\hat{Y}_t = 0 \mid Y = 1) \leq P(\hat{Z}_t = 0 \mid Z = 1).$$

Then, for any $a \geq 0$, $b\geq 0$ and $0\leq p \leq 1$, $$\begin{array}{ccl} L(\hat{Y}_t) & = & P(\hat{Y}_t = 0 \mid Y = 1) \cdot p \cdot a + P(\hat{Y}_t = 1 \mid Y = 0) \cdot (1-p) \cdot b \\ & \leq &P(\hat{Z}_t = 0 \mid Y = 1) \cdot p \cdot a + P(\hat{Z}_t = 1 \mid Y = 0) \cdot (1-p) \cdot b \\ & = & L(\hat{Z}_t) \end{array}. $$

I hope this helps.

Answer 2

Given a threshold* $t$, model 1 has lower estimated expected loss than model 2 if the corresponding ROC point of model 1 dominates** the ROC point of model 2. Here is why.

Let the confusion matrix corresponding to a particular threshold $t$ be $$ \text{Conf}_t=\begin{pmatrix} j_t & k_t\\ l_t & m_t \end{pmatrix} $$ with predicted classes in rows (row 1 ~ class 0, row 2 ~ class 1) and actual classes in columns (column 1 ~ class 0, column 2 ~ class 1). Concretely, $$ \text{Conf}_t=\begin{pmatrix} \#\{{\hat Y=0 \cap Y=0\}}_t & \#\{{\hat Y=0\cap Y=1\}}_t \\ \#\{{\hat Y=1\cap Y=0\}}_t & \#\{{\hat Y=1\cap Y=1\}}_t \end{pmatrix} $$ with $\#$ counting the number of elements that satisfy the condition. We will later add a subscript 1 for model 1 and 2 for model 2.

For any given sample, the number of actual zeros $j_t+l_t$ and the number of actual unities $k_t+m_t$ are fixed at $r$ and $s$, respectively: \begin{aligned} j_t+l_t &= r \quad \text{and} \\ k_t+m_t &= s. \end{aligned} We will make use of the latter equality in a subsequent step. Let us also define the sample size $$ n:=j_t+k_t+l_t+m_t. $$

The estimated expected loss of a model is \begin{aligned} \hat{\mathbb{E}}(L) &= \frac{1}{n}\big[ak_{t}+bl_{t}\big] \\ &= \frac{1}{n}\big[a(s-m_{t})+bl_{t}\big]. \end{aligned} Explicitly, the estimated expected losses of models 1 and 2 are \begin{aligned} \hat{\mathbb{E}}(L_1) &= \frac{1}{n}\big[a(s-m_{1t})+bl_{1t}\big] \quad \text{and} \\ \hat{\mathbb{E}}(L_2) &= \frac{1}{n}\big[a(s-m_{2t})+bl_{2t}\big]. \end{aligned}

The ROC points (specific to the threshold $t$) of models 1 and 2 have coordinates $(l_{1t},m_{1t})$ and $(l_{2t},m_{2t})$, respectively. If the former point dominates the latter point, we have $l_{1t}\leq l_{2t}$ and $m_{1t}\geq m_{2t}$ and at least one of the two inequalities is strict.

What does this imply regarding $\hat{\mathbb{E}}(L_1)$ vs. $\hat{\mathbb{E}}(L_2)$? Since $a,b>0$ and $s-m_{1t},s-m_{2t}\geq0$, then looking at the formulas above we immediately see that $\hat{\mathbb{E}}(L_1)<\hat{\mathbb{E}}(L_2)$. Thus if the ROC point of model 1 dominates the ROC point of model 2, model 1 has lower estimated expected loss than model 2.

*The relevant threshold would be the optimal one. **{ Is above and to the left } OR { is above and not to the right } OR { is to the left and not below }.