0

I have two sets (small) of features that I need to compare against a class, and decide which set is better at classifying. The features are generated using ranking methods, and only the top 2% of the features are compared, to determine which method is better differentiating amongst the top ranking features. Both the small datasets achieve a low AUC: 0.2 and 0.15 to be exact?

Could I make a conclusion that Method 1 is better than Method 2 in differentiating amongst the top 2% basis on the scores if both AUC values are below 0.5?

user48733
  • 13
bumblbee
  • 1
  • 1
  • 2
    Unless you have a different definition of AUC I think this means that they get it wrong better than chance. You need to give us more detail about your data and some summary statistics and plots for anyone to get any further with this. Edit them into the question please not as comments. – mdewey Oct 09 '16 at 12:42
  • Please register &/or merge your accounts (you can find information on how to do this in the My Account section of our [help]), then you will be able to edit & comment on your own question. – gung - Reinstate Monica Oct 09 '16 at 17:44

2 Answers2

2

To answer your question: Technically yes, the Method 1 model is better than the Method 2 model. It generates a higher probability of ranking positive instance higher than a negative instance - meaning that there's a higher chance that it actually classifies correct.

The problem is, as godspeed mentions, that anything below 0.5 is worse than classifying by flipping a coin, so both models actually suck - and it could be a sign of something going wrong in your model building.

mfvas
  • 106
  • 2
  • 7
1

Consider that if you choose the opposite of the prediction made by both classifiers, your AUC becomes 0.80 and 0.85 respectively. So my suggestion would be to check the labels on both training and test data to ensure they are not switched for some reason. Otherwise, both models should be considered "bad" as is, and the one with the smallest AUC as even worse

godspeed
  • 493
  • 3
  • 10
  • The labels are correct, it is just that the size of features used in both methods is very small. Can I make a conclusion that the first method works better in such a context? – bumblbee Oct 09 '16 at 17:48