Why do we use Linear Models when tree based models often work better than linear models?

Question

In Supervised Machine Learning, and specifically on Kaggle, it is usually seen that tree models often outperform linear models. And even in the tree-based models, it is usually XGBoost that outperforms RandomForest, and RandomForest outperforms DecisionTrees.

If this is not true, then please feel free to correct this assumption.

These are just my observations and somehow, a bunch of people share this opinion.

Why should we even use Linear models such as Linear Regression or Logistic Regression? Specifically, when they supposedly do not perform as well as tree-based models and have more requirements than tree-based models?

A similar question can be raised for tree-based models about using DecisionTrees instead of RandomForest or XGBoost.

Are there some cases where linear models should be preferred?

Answer 1

Many excellent answers already. I would add a few more aspects.

You do not say what you mean by "outperform", as in "tree models often outperform linear models". You presumably mean something like "tree models yield higher out-of-sample prediction accuracy". However:

The other answers allude to an issue that should be considered explicitly: accuracy is not the only performance dimension. There are at the very least aspects like:

• Interpretability - where linear models give you parameter estimates you can interpret
• Debuggability - which is easier for linear models than for more complex ones, related to interpretability as above - your users may ask you to improve "bad" predictions in specific cases, and it's much easier to do so for linear models
• Robustness - which may or may not be better for trees than for linear models, especially if you regularize - you may prefer lower accuracy on average as long as you get fewer "really bad" predictions
• Data requirements - as others noted, Kaggle usually has large datasets, but a real use case may have much less clean data, and simple methods may be surprisingly hard to beat especially in low data situations
• Other resource requirements - one key aspect here being expertise; a complex model requiring a lot of different data may require a data engineer and a highly qualified data scientist to get to work (but other requirements may be computing resources or something as prosaic as electricity; Petropoulos et al., 2023)
• Business value - everybody likes higher accuracy, but higher accuracy does not necessarily directly translate into higher business value (e.g., Kolassa, 2023, Foresight), and this needs to be compared to the higher resource costs per above

Data scientists naturally have a predilection for higher accuracy, and rightly so. But we do not work in a vacuum, and our models always need to be justified in the larger context, which includes other users and budgets.

Answer 2

For most problems it is easy to beat a tree model with a regression model. That's because tree models allow for all possible interactions among predictors and these are seldom needed. Interactions are notoriously hard to estimate, being double or triple differences etc., and much higher sample sizes are needed to estimate double differences (e.g., 4x greater sample sizes than needed to estimate simple differences = main effects, in an ideal case). Deficiencies in tree performance are often invisible to the user who does not assess absolute predictive accuracy of the tree or tree ensemble method. Once you get used to plotting unbiased estimates of smooth calibration trees you'll see what I mean.

An enlightening paper is this which showed that tree methods require more than 10x higher sample sizes than regression, in order to be reliable. For single tree it's much higher than that. Trees make poor use of available data and single trees can't handle continuous predictors correctly. More here.

In my experience random forests set the record for horrendous calibration accuracy (and overfitting).

Answer 3

Linear regression is also existent outside machine learning where it requires much less data. Why does Machine Learning need a lot of data while one can do statistical inference with a small set of data?

It is largely a matter of bias-variance trade-off. Kaggle problems often have large datasets that help to reduce variance and allow less biased more flexible models like tree models. On the other hand, too much variation/flexibility is neither good (leads to overfitting). If ensemble methods like random forest do better than a plain decision tree, then it is because there is too much flexibility and it needs to be tamed.

Why do 'we' still use linear models?

You still get people that try linear models because that is how Kaggle works. People with all sorts of backgrounds try to give it a go at some problem. If they end up at the bottom of the ranking then it just means that the problem requires a flexible model like a tree model: it doesn't make linear regression useless for other problems. Linear regression works very well when the true/best model is close to the solution space of a linear model.

Also related: Why do we even bother running regression models? where it is suggested to fit a plethora of variable models instead of a single one that makes sense (because we have enough data anyway).

For small problems a linear model might also be more easily interpretable. Such a situation is highlighted in the question Why would you perform transformations over polynomial regression? .

You may tackle a bunch of data with all sorts of variable models as in the image below:

But a simple model is sometimes easier to interpret, like this:

Answer 4

In addition to the excellent answers given already, a regression model yields parameter estimates, which tree models do not. If all you are interested in is prediction (which, as I understand it, is the case in Kaggle problems) then this might not matter so much. But science is about explanation, as well.

Answer 5

I share your observation that on well-behaved tabular data (quite many rows, not too many columns), a well-built boosted trees model usually beats a random forest, and a random forest usually beats a single tree.

One of the main deficits of a tree-based model over a linear model is the wigglyness/non-smoothness of the prediction function. Another one, of course, is the ease of interpretability.

One thing that I particularly like about Boosted Trees models is that you can easily modulate their complexity via tree depth (1 = additive, 2 = pairwise interactions), and interaction constraints. Combining tree-depth 2 with interaction constraints, e.g., allows to build models with some features modeled additively for maximal interpretability, while others bear pairwise interactions. Additionally, carefully set monotonicity constraints help to reduce wiggliness of the prediction function. You won't find these options on Kaggle, but they greatly improve interpretability behavior of the model.

References on constrained Boosted Trees

1. Michael Mayer and Christian Lorentzen, Lecture notes "Responsible ML with Insurance Applications" (ETH Zurich), Chapter 3
2. Python notebook with explanations in https://github.com/samathizer/boosted-trees (work in progress)

Edit

Another advantage of linear models: direct availability of inferential methods, even if they require additional assumptions to be sufficiently met.

Answer 6

You can view xgboost as a stepwise linear regression adding nonlinear inputs.

So it's more a question of how complicated you want to make the model

Answer 7

To add to mentions of interpretability concerning linear models, this tends to open the door more for analytical explanations--equations connected to other equations and constants through theoretical reasons, as happens in like physics, theory explains, experiments observe, within error, e.g. V = IR.