Classification accuracy on multi classes

Question

My target has 5 classes. My testing dataset has an accuracy of about 34%. Can I assume this is a reasonable model purely based on classification accuracy, since random guessing is 20%.

Answer 1

Not necessarily, that assumes that the 5 classes are equally distributed. Consider the case of imbalanced data where 90% of the data is class 1 and the rest of the data is split between classes 2-5. If we were to predict every single observation as being class 1 we could expect accuracy of 90%! This is clearly not a useful model demonstrating the importance of looking at other metrics of performance such as precision, recall, ROC etc. Accuracy can still be considered but the minimum baseline should be performing better than the majority class.

Edit: Just noticed Djib2011 beat me to the same point in the comments.

Answer 2

Just to add slightly to @SerafFej's answer. A competent practitioner will take the relative class frequencies into account when assessing the accuracy. However, a general audience may not appreciate that subtlety, so it is incumbent on the practitioner not to simply present the accuracy, but put it into context by stating the accuracy of the baseline classifier that just predicts the most common class, ignoring the attributes. Alternatively, you could use something like Cohen's kappa statistic instead:

$\kappa = \frac{Acc - Acc_{random}}{1 - Acc_{random}}$

This tells us the proportion of the possible improvement in accuracy that we have obtained by using the attributes. So if the classifier is only guessing, $\kappa$ will be zero and if the classifier is perfect, $\kappa$ will be 1. Which makes the "added value" of the classifier much more apparent to a general audience. This statistic was being widely used in the neural network community back in the 1990s, but not under the same name (just an obvious solution to the problem).

Note however it is just an affine transformation of the accuracy, so it isn't telling us anything that accuracies do not already tell us. Ranking classifiers by accuracy and by $\kappa$ will give the same ranking.

Can I assume this is a reasonable model purely based on classification accuracy

The other problem here is that we can't tell whether the model is reasonable if we don't know how separable (easy) the classification task is (i.e. the accuracy of the Bayes optimal classifier). If we have a very easy classification problem where all of the classes are well separated from each other 34% may be an extremely poor level of performance. On the other hand, if there is a substantial overlap between classes, then it may be that it is very difficulty to improve on a classifier that just picks the most common class all the time. See my question here which gives an example where the "default classifier" that ignores the attributes entirely is optimal.

Unfortunately if we had a means of knowing the Bayes error rate, we probably would have no need to construct a classifier from data, so this end of the scale is not so easy to establish.