13

Let's say I'm comparing 60 different model hyperparameter value combinations using 10-fold cross-validation. It's tempting to simply select the hyperparameter combination whose mean accuracy is highest across the folds. However, should one make use of the standard deviation of the accuracies when deciding on the best hyperparameter combination? If so, any particularly rule of thumb (e.g. go with the hyperparameter combination that has the highest mean accuracy amongst the better half in terms of standard deviation)

blu
  • 533

2 Answers2

12

Sort of. There is the so-called "one standard error rule," which does use the standard deviation of the prediction error estimates, although not in quite the way you mentioned: instead you divide the standard deviation by the square root of the number of estimates to form the standard error of the mean estimate.

The one standard error rule says: pick the simplest model whose mean estimated prediction error is within 1 standard error of the best-performing model's estimated prediction error. In practice, the "simplest model" usually means "the most strongly regularized model." And of course, the "best-performing model" is the one with the lowest mean estimated prediction error of all models tested.

Stated a bit more plainly, the rule says that we want to pick the simplest model that still does essentially as good a job as the best-looking model -- the best-looking model could be far more complicated, despite only a marginal increase in performance.

Jake Westfall
  • 12,557
  • 1
    Thank you. That's very helpful. In the context I'm describing, what does standard error refer to? For instance, do you mean computing sd(x)/sqrt(length(x)) where x is the set of 10 accuracies (one per fold) corresponding to the best-performing model? – blu Aug 12 '16 at 02:17
  • I am guessing that this answer only applies if the data is sampled uniformly from the same population. However, if the data was sampled from different groups, and group k-fold validation is performed, then the variance in CV prediction error could come not just from the model but from the variance between the groups, i.e. the variance between the validation sets for different folds. Just something to keep in mind. – rinspy Jun 07 '17 at 07:49
  • Jake, I was under impression that "standard error" refers to standard error across CV fold, i.e. standard deviation of the estimates divided by the square root of the number of folds. Why are you saying that it's simply standard deviation? Can you point to any reference? Do you know what convention e.g. glmnet is using? – amoeba Feb 02 '18 at 10:54
  • 1
    @amoeba Ah, yeah, you're right. I guess I had a brain fart when I wrote that. I'll fix it. Thanks! (I guess for some reason I had been interpreting it as being the SE of the estimates themselves, not the SE of the mean estimate -- the latter of which would of course require dividing by the total number.) – Jake Westfall Feb 02 '18 at 14:19
  • 1
    @blu It refers to the standard error of the mean CV estimate, so yes, it would be computed as sd(x)/sqrt(length(x)) in R. – Jake Westfall Feb 02 '18 at 14:23
1

In addition to @jake-westfall's answer which gives you an easily applied rule of thumb:

The variance you observe between the accuracies/errors of the different folds is composed (at least) of some variance due to the limited number of cases tested by that surrogate model and some variance due to the variations between the surrogate models (instability). The latter is the variance you want to trade off against bias with your regularization, while the former hampers your ability to detect improvements. So for a second look, I recommend to check that the variance due to the number of tested cases is low enough to sensibly allow the comparison.

cbeleites unhappy with SX
  • 38,684