One class never gets predicted, regardless of the model

Question

I'm working on a classification problem from a dataset containing three classes, with proportions {"0":0.43, "1":0.25, "2":0.30}. However, whenever I train a model, it never predicts class "1" (literally ZERO predictions).

Even weirder, this happens regardless of the model I train (tried the usual, SVM, RF, tree boosting, etc.). So I guess there is something in the data that I did not catch, but the fact that there is not even one or two predictions for class 1 is really disturbing.

Attached is the distribution of my test set, vs. the distribution of my predictions (here for a baseline SVM) :

Thanks.

Answer 1

Let's take a much simpler example that might lead to that kind of result, using this frequency table roughly based on your charts:

$$\begin{array}{cccccc} & & \text{feature:} & A & B \\ &\text{class:} & & & \\ & 0 & & 1050 & 20 \\ & 1 & & 550 & 100 \\ & 2 & & 150 & 600 \\ \end{array}$$

A model trained on this data is likely to suggest when observing a new case of $A$ that class $0$ would be the best prediction, while when observing a new case of $B$ that class $2$ would be the best prediction. So it would never give a prediction of $1$. These predictions would be correct the majority of the time, and no other approach would be likely to do better (unless you are allowed to make non-categorical class predictions like $0.4$ or $1.7$).

Your actual model is likely to be more sophisticated than this, but if a particular class is always less common than one of the other classes no matter what additional information you condition on, then you could see a similar failure to ever predict that class.

Answer 2

There is nothing particularly strange about this, it may well be that the model is giving the optimal answer to the question it has been posed. If the density of patterns belonging to the minority class (weighted by its prior probability) is less everywhere than the weighted density of the majority class, then no pattern is ever more likely to belong to the minority class than the majority class. In that case, the optimal accuracy is obtained by assigning all patterns to the majority class.

Here is a simple two-class example, with Gaussian class conditional densities:

If this is not a satisfactory answer, it usually means that the misclassification costs are not equal (in your case classifying a class 1 pattern as a class 0 or class 2 is a worse error than the other way round). The solution is to work out what the matrix of misclassification costs should be and use "minimum risk classification". If you have a probabilistic classifier, that can be done after training. If you use a discrete classifier, like the SVM, then the misclassification costs must be built in at training time (for the SVM having different values of C, the regularisation parameter, for each class).

Most classification methods are set up by default to assume that misclassification costs are equal, but that is not that often the case in real world classification tasks. Cost-sensitive learning is something that practitioners need to have in their toolbox.

Answer 3

As Henry noted (+1) this is a natural consequence of model fitting if the classifications are clearly discriminatory. A by-product of that is that the less likely events will get predicted wrong. You do not want perfect predictions in your model, as this is often a sign of overfitting.

On another note, it is much more important to look at out-of-sample predictions than in-sample predictions. Your model will usually correctly predict things more often on the data it was trained on because this is how it was informed. Testing on out-of-sample data is much more valuable for seeing how accurate your model is in the real world.