13

When coding categorical features for linear regression, there is a rule: number of dummies should be one less than the total number of levels (to avoid collinearity).

Does there exist a similar rule for Decision Trees (bagged, boosted)? I am asking this because a standard practice in Python seems to be to expand n levels into n dummies (sklearns' OneHotEncoder or Pandas' pd.get_dummies ) which appears suboptimal to me.

What would you suggest as best practices for coding Categorical features for Decision Trees?

Sergey Bushmanov
  • 1,197

2 Answers2

13

It seems like you understand that you're able to have n levels, as opposed to n-1, because unlike in linear regression you don't need to worry about perfect colinearity.

(I'm coming at this from an R perspective, but I assume it's the same in Python.) That depends on a couple of things, such as 1) which package you're using and 2) how many factor levels you have.

1) If you are using R's randomForest package, then if you have <33 factor levels then you can go ahead and leave them in one feature if you want. That's because in R's random forest implementation, it will check to see which factor levels should be on one side of the split and which on the other (e.g., 5 of your levels might be grouped together on the left side, and 7 might be grouped together on the right). If you split the categorical feature out into n dummies, then the algorithm would not have this option at its disposal.

Obviously if the particularly package you're using can't handle categorical features then you'd just need to create n dummy variables.

2) As I alluded to above, R's random forest implementation can only handle 32 factor levels - if you have more than that then you either need to split your factors into smaller subsets, or create a dummy variable for each level.

Tchotchke
  • 1,014
  • 2
    Thanks! Am I getting you correctly: unless I'm modelling in R, where categorical features in randomForest are coded automatically, I should go with n dummies because collinearity is not a problem for RF? – Sergey Bushmanov Oct 20 '15 at 19:47
  • 3
    Having more than 32-level binary-encoded categories will have slightly different behavior in the tree, since RF will just select from among those binary columns, rather than selecting the single column of the factor with many levels. This subtle difference means that the split on the binary columns will be less informative compared to splitting on the factor column, since there's only one choice (0/1) versus (1/{2,3}, {2,1}/3) etc. – Sycorax Oct 20 '15 at 19:47
  • @user777 It's not a problem of having over 32 variables. It's a problem of not having "grouped" category variables in Python sklearn... Practically speaking, is there an evidence (practical experience, research, etc) that "dummified" variables will perform worse than "grouped" categorical variables [in R] – Sergey Bushmanov Oct 20 '15 at 19:58
  • 1
    Theoretically, we could expect the non-grouped to perform slightly worse, because you're giving the model less flexibility. In the grouped case, if it were truly better to treat that feature as not grouped, then the model would be able to do that (by putting one group on one side, and then all of the rest on the other). However in practice, I would be surprised if there were much difference (particularly in the case of RF, where you are creating so many trees) – Tchotchke Oct 20 '15 at 21:13
  • 2
    I find the randomForest implementation using features with many factor levels (>15) slow as mentioned, but also to yield mediocre model performance. I guess a very high number of possible splits will reduce the intended decorrelation of trees in the ensemble. extraTrees and Rborist only try a subsample of catagorical splits in each node. That may help decorrelation and certainly speed. Thus a range of solutions between "randomForest try any split" and "sklern dummy-variable only try 1-vs-rest splits" are possible. Also diffeent clusterings of the many levels into fewer levels may show useful. – Soren Havelund Welling Oct 21 '15 at 13:13
4

There's another approach to dealing with categorical variables that is called target/impact encoding.

In this scheme the idea is to encode the feature using a single float column in which the value is the average of the target variable over all rows that share the category. This is especially useful for tree based models since it imposes an order relationship within the feature (ie values to the right of the category have higher mean response than values to the left) and it makes it easier to split the predictor space.

Here's a nice explanation of the subject:
https://towardsdatascience.com/why-you-should-try-mean-encoding-17057262cd0

And here's a link to the paper that originally proposed the encoding: http://helios.mm.di.uoa.gr/~rouvas/ssi/sigkdd/sigkdd.vol3.1/barreca.pdf

There's some more details to avoid estimating the mean in categories with low counts and also there's another model, CatBoost, proposing a solution to the biasing introduced by this encoding, but in my experience it's a simple and very useful way to encode high cardinality categorical variables.

Ludecan
  • 141