0

I am trying to understand how xgboost prediction evolves by adding more trees (of course the ROC AUC will increase). But to my surprise, when there are small number of trees in the model, the average prediction for the entire training sample is always around 0.5, although my training data average of target variable is 0.1. (I also switch 0, and 1 of my target variable, such that the average becomes 0.9. The average prediction is still 0.5).

Could you explain why the average prediction is not starting somewhere close to the average of target variable? I understand this is not regression, and we don't have an intercept. But still this result seems not intuitive.

In addition, adding more trees (or increase learning rate) will shift the prediction average towards the observed average. Does it mean there is a minimum number of trees needed in the model? Otherwise, the prediction average is not aligned to the observed average.

Any comments are very well appreciated. Thank you in advance.

mkwargs = {'learning_rate':0.01, 'reg_alpha':1.0, 'reg_lambda':100.0, 'gamma':1.0,
           'objective':'binary:logistic', 'subsample':0.8, 'colsample_bytree':0.8}
Watchung
  • 307
  • 1
    I think this question covers the same point you raise: https://stats.stackexchange.com/questions/611958 in particular see my comment to Ben in Dave's answer. Also remember that XGBoost "always does regression". So it still starts with 0.5. I cover an example of that here: https://stats.stackexchange.com/questions/395697 – usεr11852 Apr 28 '23 at 01:57

0 Answers0