Feature importance with scikit-learn Random Forest shows very high Standard Deviation

Question

I am using scikit-learn Random Forest Classifier and I want to plot the feature importance such as in this example.

However my result is completely different, in the sense that feature importance standard deviation is almost always bigger than feature importance itself (see attached image).

Is it possible to have such kind of behaviour, or am I doing some mistakes when plotting it?

My code is the following:

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier()
clf.fit(predictors.values, outcome.values.ravel())

importance = clf.feature_importances_
importance = pd.DataFrame(importance, index=predictors.columns, 
                          columns=["Importance"])

importance["Std"] = np.std([tree.feature_importances_
                            for tree in clf.estimators_], axis=0)

x = range(importance.shape[0])
y = importance.ix[:, 0]
yerr = importance.ix[:, 1]

plt.bar(x, y, yerr=yerr, align="center")

plt.show()

Answer 1

You are using RandomForest with the default number of trees, which is 10. For around 30 features this is too few. Therefore standard deviation is large. Try at least 100 or even 1000 trees, like

clf = RandomForestClassifier(n_estimators=1000)

For a more refined analysis you can also check how large the correlation between your features is.

Answer 2

Your result is not that weird. As lanenok states, you should in a first step increase the number of trees in order to make sure that you get a 'statistical' result concerning the feature importances.

However, as this paper by Genuer et al. (2010) shows, you can actually use the standard deviations in order to eliminate features. To quote: "We can see that true variables standard deviation is large compared to the noisy variables one, which is close to zero. "

Answer 3

Try clf = RandomForestClassifier(max_features=None). The max_features param defaults to 'auto' which is equivalent to sqrt(n_features). max_features is described as "The number of features to consider when looking for the best split." Only looking at a small number of features at any point in the decision tree means the importance of a single feature may vary widely across many tree. So, don't look at a random subset, just look at all features at every level of the tree.

Answer 4

A common reason for this is that the parameters you supplied (or defaulted) to RandomForestClassifier are not suited for your dataset.

A common way to address this problem is to search the hyperparameter space using e.g. GridSearchCV:

from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score, make_scorer

param_grid = {'n_estimators': [10, 100, 1000], 'max_features': [5, 10, 20, 30]}
clf = GridSearchCV(RandomForestClassifier(), param_grid, cv=5, scoring=make_scorer(accuracy_score))

param_grid here is the permutations of parameters that you want to search in, and the make_scorer(accuracy_score) is the measure you want to optimize.

Note that accuracy_score is suitable for balanced sets, but not for unbalanced sets. Choose a suitable metric to on your particular objective.

Answer 5

There could be multiple reasons. The number of trees and the depth can change your results. If your model doesn't perform well after selecting the parameters (cross-validation etc.), it's probably because your features are not very predictive, so they get picked almost "randomly" which leads to high standard deviations from tree to tree. But there are other possibilities, e.g. it could also be that your features are highly correlated. A little more information would be helpful.