1

I can't solve the problem about the AUC ROC metric. Problem condition: on the answers (estimates) of the algorithm, objects of class 0 are distributed uniformly on the segment [0, 2/3], and answers of class 1 are distributed uniformly on the segment [1/3, 1]. What is AUC ROC equal to?

My approach. AUC is the area under the curve that can be expressed using TPR and FPR:

$$\int_0^1 TPR(x)\cdot FPR'(x)\, dx$$.

Since the distribution is uniform, TPR and FPR can be represented as: $$ TPR(x) = \begin{cases} 0, & \text{if $x$ < 1/3} \\ \frac{3}{2}\cdot(x-\frac{1}{3}), & \text{if 1/3 $\le$ $x$ $\le$ 1} \end{cases} $$

$$ FPR(x) = \begin{cases} \frac{3}{2}\cdot x, & \text{if 0 $\le$ $x$ $\le$ 2/3} \\ 1, & \text{if $x$ > 2/3} \end{cases} $$

Then we get the integral: $$\int_\frac{1}{3}^\frac{2}{3} \frac{3}{2}\cdot(x-\frac{1}{3})\cdot \frac{3}{2}\, dx = \frac{1}{8}$$.

But the answer is $\frac{7}{8}$. Which means that either I made a mistake in the calculations or the classifier predicts zeros ($1 - \frac{1}{8} = \frac{7}{8}$). Where did I go wrong?

rep_ho
  • 7,589
  • 1
  • 27
  • 50
Fissium
  • 111
  • An ROC AUC should never be less than 0.5. If it is, your classifier is doing worse than random, and you should flip the predictions. From the problem statement, it's clear that Class 1 samples generally score higher than Class 0 samples, so the AUC should definitely be above 0.5 – Nuclear Hoagie Feb 05 '20 at 15:42
  • @NuclearWang that's not true, you cannot change your predictions after you saw the results. 0.5 is chance level, that means that sometimes you have to be able to make bellow chance level predictions. – rep_ho Feb 05 '20 at 16:09
  • @rep_ho I'm not saying that it's impossible to get an AUC below 0.5, but that model is complete garbage that would be outperformed by random class assignment. In my experience, an AUC that is statistically significantly below 0.5 is strongly indicative of an error in the problem formulation or AUC computation. An AUC<0.5 indicates there's enough signal to predict well, but for some reason you're not. There's no reason a supervised method should consistently learn the opposite of what you're trying to teach it. See https://stats.stackexchange.com/questions/266387/can-auc-roc-be-between-0-0-5 – Nuclear Hoagie Feb 05 '20 at 16:26
  • @NuclearWang I added my answer there – rep_ho Feb 05 '20 at 17:22

1 Answers1

1

Since this is a self study question I will give you some hints, instead of the complete answer.

I don't know calculus, so we have to do it logically. First, having uniformly distributed predictions on the range [0, 2/3] and [1/3, 1] is for the sake of AUC the same as having 2 predictions for each class, let's say

P_A = 1, 2 and

P_B = 2, 3.

Method 1, concordance probability: As you know the definition of AUC is the probability that the randomly selected prediction from the group A will be ranked lower than the randomly selected prediction from the group B. Or in other words, proportion of pairs of predictions where A is lower than B. So we have in total 4 possible pairs, in how many pairs is A lower than B (Ties counts as 0.5)?

Method2, rank-sum: AUC is just a simple transformation of the Mann-Whitney U statistic wiki link. To use this formula, we have to change the predictions to ranks, again, ties counts as 0.5 so ranks of our predictions are

R_A = 1, 2.5

R_B = 2.5, 4

The u statistic is sum of ranks for one class - the best possible sum of ranks. So for R_A that would be U_A 1 + 2.5 - 1 + 2 = 0.5. Formula for AUC (or 1-AUC, depending on which way you rank your data) is U_A/U_A+U_A or U_A/n1n2

See also How to calculate Area Under the Curve (AUC), or the c-statistic, by hand

Edit: I think your integral is just other way around, if lower is better for your first class (which it should be if your aims is to get 7/8), than your TPR should start at 1 and not 0, since everything bellow the threshold is correct

rep_ho
  • 7,589
  • 1
  • 27
  • 50