I have crossvalidated my models and measured RMSE between the modelled values and reality:
RMSE <- function(err) sqrt(mean(err^2))
RMSE(predicted - reality)
I am going to choose model with lowest RMSE. But I would like to know what difference in RMSE is still significant. I guess I could do some F-test comparisons, but as I have tens of models I don't like the idea of pairwise comparisons. This leads me to question
How can my RMSE function be extended to compute SE of RMSE?
How to do this in R? I guess this will be somehow based on Chi-square distribution parametrized by length(err) and maybe somehow scaled by sd(err) (?), but I don't know how to make this statistically correctly in R.
