2

For regression models we have the AIC to use as a quality metric. It has a high score for models that use fewer exogenous variables compared to models with many exogenous variables (holding the variance explained constant).

Decision trees also can suffer from the same plight, being too complicated, perhaps because of over-fitting. Is there anything like the AIC for decision trees? Is there a way to adapt the AIC to a decision tree? If not, what should we do? Perhaps find a way to relate the number of exogenous variables to the purity ratios?

Arash Howaida
  • 857
  • I haven't read through it all, but this might include what you're looking for: https://www.cs.cmu.edu/~aarti/Class/10701/recitation/decisiontree_modelselection.pdf – R Greg Stacey May 17 '17 at 21:39
  • @Qroid Wow, that PDF has a very interesting worked out example... haha. It does have what I was looking for, thank you! I hope someone can still answer the question in an intuitive way. I eventually got lots in the PDF's formal equations. – Arash Howaida May 18 '17 at 05:09

1 Answers1

1

It would be elegant to have some kind of AIC/BIC-style penalty on a decision tree. However, the typical way to penalize a machine learning model for being overly complex is to try it on some data that were not used to develop the model, with the thinking being that, if the model is so complex that it allows for fitting to the noise (coincidences in the data), the model will not have such a great fit on the new data and will have a poor score. When you see something like a train/test split, the test data are being used for this very evaluation.

There are drawbacks to such a strategy. For instance, what if you have some good or bad luck with your holdout data and just get one score that is particularly high or low? Cross validation is a possible remedy for this. As another example, having a holdout set limits the previous data available for training. Bootstrap validation is a possible remedy for this.

Dave
  • 62,186