1

For my machine learning study, I tested different algorithms like SVM, SMO, Naive Bayes, Trees etc. All the algorithms resulted with low accuracy levels. In fact the highest accuracy I obtained was 46% using Naive Bayes.

Then I tried to do a feature selection. I used InfoGainAttributeEval in WEKA to do this. It ranked 7 features out of 27 I used, and then, I tried the classification with those 7 features only. But, it resulted with worse accuracy levels. The accuracy of all the algorithms other than SMO got decreased. Naive Bayes resulted with 36% of accuracy.

As I have heard and learned, feature selection is for decrease the complexity and improve the accuracy. But in my case, Why it decreased the accuracy also?

vigamage
  • 293
  • "As I have heard and learned, feature selection is for decrease the complexity and improve the accuracy" - This is not necessarily true. While feature selection does decrease complexity, it does NOT have to improve accuracy. Using automatic feature selection is probably not your answer. Try out manual approaches to understand which factors are most strongly related and any domain knowledge to come across any candidate models. However, if your predictors inherently are weakly related to your target, you aren't likely to get a good model. – Arun Jose Dec 07 '15 at 07:15
  • yes true, my research finding is going to be the features I am considering cannot address my question. BTW can u provide a paper I can refer to mention in my thesis that "feature selection does not have to improve accuracy" – vigamage Dec 07 '15 at 08:12

1 Answers1

1

As I have heard and learned, feature selection is for decrease the complexity and improve the accuracy.

This is a common way to think that seems to follow the following logic.

  1. Including many features gives the model considerable flexibility.

  2. That flexibility puts us at risk of overfitting and achieving pitiful generalizability that keeps models from making useful predictions on new data (when we’re truly interested in the predictions).

  3. Therefore, reduce the feature count to reduce the overfitting potential.

It is true that reducing the feature count helps quell overfitting concerns. However, leaving out features deprives the model of the unique information contained in that feature, information that might be a critical determinant of the outcome. Thus, while your feature selection probably quells overfitting concerns, it does so at the risk of introducing underfitting concerns.

You seem to have underfit your data by depriving the model of useful features for making predictions, leading to the decrease in accuracy (setting aside the known issues with accuracy). Because this can happen, not every statistician is such a fan of feature selection.

Dave
  • 62,186