In this post I established that the standard error of cross-validation prediction error is the standard deviation of prediction error across folds divided by the square root of the number of folds. Hence in a lasso regression with 100 candidate values of lambda there will be 100 standard error of prediction errors.
This brings me to my next question. There is a convention (see here and here) that one should report the results of the simplest model within one standard error of the model with the minimum out-of-sample prediction error. As I mentioned I established that with 100 candidate values of lambda there are 100 standard errors. So my question is which standard error do we use to make this judgement?
The figure below plots average out-of-sample prediction error against 100 candidate lambda regularisation parameters in a cross-validation procedure I ran. The red square is the value of lambda that yields the minimum average cv error (called in the package I am using via fit$lambda.min) and the green triangle is the value of lambda for the simplest model one se from the minimum (called via fit$lambda.1se).
But there are 100 standard errors, one for each model. So which se is used to select the 1se model? The se for the minimum cv error model?
