What to do for AUC less than 0.5?

Question

I've trained a Random Forest model on a dataset of 60 protein predictors for healthy controls (label 0) and cancer patients (label 1).

I then tested this model on a dataset of at-risk patients divided into those who later got cancer (label 1), and those who didn't (label 0).

My model's performance gave an AUC-ROC of 0.4.

Other threads and papers (linked below), say that for AUC < 0.5, a classifier has useful information but is applying it incorrectly. People seem to suggest reversing the labels, to give an AUC-ROC of 0.6 Can AUC-ROC be between 0-0.5 http://people.inf.elte.hu/kiss/13dwhdm/roc.pdf

However, would this be appropriate in this case? Reversing the test dataset labels would mean giving the at-risk individuals who stayed healthy a label of 1 (the same as the cancer patients in the training data), which doesn't seem correct to me??

Answer 1

"Reversing" the AUC by taking AUC = 1 - AUC would be appropriate if you had no a priori information about whether to expect larger or lower values for the positive group. For instance if you were measuring a molecular biomarker, it could be present with a decreased concentration in the cancer patients. Unless and until you know more about it, you can absolutely reverse it.

However it is not your case. You trained a model to detect cancer patients. I assume that you are probably obtaining as an output the probability that the patient belongs to the cancer group. This probability has to be higher for the cancer group, otherwise you have a problem.

What you are looking at is a confounding factor that you haven't identified yet. You model learned to identify risk factors that weren't present in the healthy group, but are now in your "at risk" group, just as they were in the "cancer" group or even more strongly so.

Inference is hard and your model just isn't quite good enough at it. Finding differences to a "healthy" group is easy in my experience, although usually quite useless in practice. In the future, try to collect a training sample that is as close to your target clinical question as possible. Until then, please do not state that your AUC = 0.6.

Answer 2

How large are your training and test samples ? You have to keep in mind that, when training a classifier, there is some variance over the estimation of your model and you may achieve an accuracy which is lower than one of a constant classifier (or an AUC which is lower than 0.5). If you work with a small sample, this is even more likely.

Edit.

The training dataset is 300 cancer patients and 130 controls. The test dataset is 79 who stayed healthy and 20 who later got cancer

Given this information, the training data set has much less cancer patients than the test set (in terms of ratio).

Most learning algorithms rely on the fact that training and testing set have a similar distributions (of the target variable, and the features). Here, this assumption is violated (since having a similar distribution would imply having the same ratio of positive examples in the target value).

However, this does not justify "reversing" the predictions.

Imagine the following scenario (in the case of a linear model) : $\hat{y}(x)= a + b\tanh(x)$. $\hat{y}$ is the probability of an event, $a$ is the probability of this event given $x=0$ and $b$ describes the impact of a certain predictor $x$.

If, for some reason, this classifier performs poorly and $\tilde{y}=1-\hat{y}$ yields better result, then note that $\tilde{y}(x)= (1-a) - b\tanh(x)$

Now the conclusion about the role of $x$ is reversed as well! If $b>0$, then higher $x$ increase the likelihood of a cancer. But this conclusion does not hold any more !