10

In all the (regression) random forest papers I've read, when it comes the time to gather the predictions of all the trees, we take the average value as the prediction.

My question is why do we do that?

Is there is a statistical justification for taking the average?

EDIT: To clarify the question, I know it's possible to use other aggregation functions (we use the mode for classification), I'm mostly interested in whether there is some theoretical justification behind the choice of the average function.

Firebug
  • 19,076
  • 6
  • 77
  • 139
Bar
  • 2,862
  • 1
    Possibly relevant: https://stats.stackexchange.com/questions/174390/why-glms-predict-the-mean-and-not-the-mode – Matthew Drury Oct 05 '17 at 19:34
  • enter image description here Here's the reference: On the probabilities vs class labels part http://sebastianraschka.com/Articles/2014_ensemble_classifier.html#2-prediction-based-on-predicted-probabilities-equal-weights-weights111 – PauAI Feb 13 '18 at 22:53

6 Answers6

6

When using the average, you are saying two things:

  1. Outliers are not a huge problem (otherwise you would use the median or at least filter out some outliers before taking the average)
  2. Every prediction has the same weight (otherwise you would factor in weights)

You shouldn't expect there to be huge outliers since you can make the sample size big enough for them to matter less in the average and since you would expect a minimum of stability from the predictions of the individual trees.

There is no reason to think some trees should have more predictive weight than others, nor a way to determine such weights.

You cannot really use mode since the predictions are on a continuous scale. For example, if you had the predictions 80 80 100 101 99 98 97 102 103 104 96, mode would predict as 80. That cannot be what you want. If all values have distinct decimals, mode wouldn't know how to decide.

Other averages than the arithmetic mean exist, like the geometric mean and the harmonic mean. They are designed to pull the average down if there are some low values in the series of data. That's not what you want here either.

David Ernst
  • 3,151
  • 10
  • 15
  • 1
    If outliers is a concern, there are alternatives in between the median and the sample mean, like winsorized or trimmed means, which could give good outlier protection while being more efficient than the median. – kjetil b halvorsen Oct 05 '17 at 13:35
5

I've always thought about the averaging in terms of the bias-variance tradeoff. If I remember correctly Leo Breiman hinted at this in the randomForest paper with his statement "... are more robust with respect to noise."

The explanation goes like this: basically you are taking a bunch of trees that are grown to full length-no pruning-so you know they will each be biased by themselves. However, the random sampling that induces each tree in the forest should induce under-bias as often as over-bias. So by taking an average you then eliminate the bias of each tree-the over+under biases canceling. Hopefully in the process you also reduce the variance in each tree and so the overall variance should be reduced as well.

As indicated by the other answers to the post, this might not be the only reason for averaging.

Lucas Roberts
  • 4,259
  • 1
    Accepting this, as the answer seems that the average is chosen out of "intuition" rather a particular theoretical motivation, contrasting to the theoretically motivated answer on GLMs: https://stats.stackexchange.com/q/174390/16052 – Bar Oct 06 '17 at 14:28
  • 1
    @Bar, with decision trees, the problem is really that global optimization is NP-hard so greedy optimization is done. The greedy optimization for each tree doesn't tell us about the forest. Unfortunately, the math for this problem is less developed than either of us would like. – Lucas Roberts Oct 06 '17 at 14:56
3

Of course you could use any aggregation function that is useful in your particular situation. The median is a good way of making a small sample robust against outliers. In regression forests you can usually influence the sample size to avoid having the problem of small sample sizes. Thus the mean seems sensible in a very large fraction of use cases.

Bernhard
  • 8,427
  • 17
  • 38
1

Wouldn't it be possible as well to take the median, mode, or some other aggregate function?

Random Forest classification (i.e. not probability estimation) is based on the mode of the predictions (majority voting), so yeah, you can aggregate the results as you like.

Firebug
  • 19,076
  • 6
  • 77
  • 139
  • Thanks for the answer, I've added a clarification to my question. I know it's possible to use other aggregation functions, what I'm wondering is if there's any theoretical reason for the choice of average. – Bar Oct 05 '17 at 11:39
1

First things first. As many other people said you can use other metrics but the average is the "default" option.

As a default option, one would set a function that works under some mild conditions

Now, If you think about it, a random forest is a collection of trees and each of these trees has the objective to estimate your numeric response variable.

Additionally, as @David Ernst correctly mentions:

There is no reason to think some trees should have more predictive weights as others, nor a way to determine such weights.

Furthermore, there is no reason to think that these trees will have different standard deviations. Again, under mild conditions!

That being said, the average should work because of the Weak law of large numbers

Vasilis Vasileiou
  • 1,423
-1

In ensemble. Averaging is prioritizing more on confidence rather than majority.

Example you have 3 trees,

2 of them vote A with 22% confidence and 1 voted B with 90% confidence.

If we use majority we get vote A. Average of 22, N, N If we use confidence we get vote B. Average of 90, N, N

It would make sense to go with the 90% confidence since its more sure than the majority of others with only 22% confidence.

PauAI
  • 101
  • 2
  • It is not quite clear to me what your point is. Can you edit to clarify it perhaps? What would happen if there were 100 A and only one B with the same set of confidence ratings? – mdewey Feb 12 '18 at 15:51