3

I have the following output from a boosted regression trees model and I would like to calculate the total deviance explained. 

mean total deviance = 1.283 
mean residual deviance = 0.107 

estimated cv deviance = 0.212 ; se = 0.045 

training data correlation = 0.97 
cv correlation =  0.937 ; se = 0.016 

training data AUC score = 1 
cv AUC score = 0.996 ; se = 0.002

I have come across two methods to do this which give me a different answer.

1) D2 = 1 – (residual deviance/total deviance) (Nieto and Mélin, 2017)

With my results this equation is

D2 = 1 - (0.107/1.283) = 0.92

2) D2 = (total deviance - cross validated residual deviance)/total deviance (Leathwick et al., 2006)

With my results this equation is

D2 = (1.283-0.212)/1.283 = 0.83

The paper for method 2 does not provide the equation in their text but it is clear given their model results (Table 3, pp.272) that this is how it was calculated. Also, there is a question here that discusses this method.

Which of these methods is correct?

Jo Harris
  • 103
  • Just to be clear, in the second equation, what is denoted as residual deviance is what, in the first question, is denoted as total deviance. Can you please clarify that these are meant to be the same (i.e. the total deviance)? – usεr11852 May 17 '19 at 09:20
  • My apologies for the mistake. I have now edited my question. – Jo Harris May 18 '19 at 10:12
  • Thank you for amending that. Good on you to catch that up! (+1) Please see my answer below for more details. – usεr11852 May 22 '19 at 00:48

1 Answers1

3

Neither of the two methods is "wrong"; they measure different aspects of the data-modelling aspects. That being said, I would have a strong preference on the Leathwick et al. approach as it tries to explicitly account for sampling variability; ultimately it is a repeated cross-validation report of our performance metric.

To comment on this a bit further: On the one hand, the Nieto & Mélin's approach aims to directly generalise the concept of coefficient of determination $R^2$, using the deviance residuals instead of the actual ones. It does not resample the data or anything similar, it directly reports the choose metric (here $D^2$) overall the whole data. This can potentially lead to unreasonably optimistic results regarding the generalisation of our model's performance. On the other hand, the Leathwick et al. approach aims to incorporate the sampling variability directly through the repeated cross-validation step. We estimate values of our performance metric using "unseen" data that were excluded during training. Note that the "optimistic bias" can be immediately seen in the Table 3 of the L. et al. paper, if we use the model residual deviance instead of the CV residual deviance. In that case for example the CV-generated $D^2$ would move from $0.600$ to $0.663$ for the case of a Boosted Regression Tree with tree size 5. The sample size of your particular application is not explicitly stated. Nevertheless given it is not gigantic, reading through the methodology presented in Beleites et al. (2013) Sample size planning for classification models is a good starting point to get idea of how to assess sample size consideration in a (multiple) CV procedure.

usεr11852
  • 44,125