Significance Test for Comparing RMSE values

Asked Sep 08 '20 at 17:55

Active Sep 08 '20 at 18:14

Viewed 288 times

I want to use the RMSE measurement to compare 3 statistical regression models against a baseline model. I want to do this based on a null hypothesis testing framework where I train the models using n-fold cross-validation and perform a significance test.

My question is, which significance test should I use in the RMSE case?

edited Sep 08 '20 at 18:14

asked Sep 08 '20 at 17:55

MarG

What is your null hypothesis? – Dave Sep 08 '20 at 18:11
@Dave That there is no statistical significance between the RMSE of the baseline model and each of the other models RMSE values – MarG Sep 08 '20 at 18:15
So something like ANOVA but first RMSEs instead of means? (As a heads up, “statistically significant” is never part of a null hypothesis.) – Dave Sep 08 '20 at 18:21
@Dave I was thinking that I would need to use a Wilcoxon sign test? – MarG Sep 08 '20 at 19:01
With one observation per model? – Dave Sep 08 '20 at 19:13
@Dave If I use n-crossvalidation I would have n observations per model? – MarG Sep 08 '20 at 19:22
Then you have a violation of independence. – Dave Sep 08 '20 at 19:26
@David Do you perhaps have another suggestion? :) – MarG Sep 08 '20 at 19:39
I've had similar thoughts but never did pursue the PRESS statistic that whuber mentioned: https://stats.stackexchange.com/questions/452453/regression-hypothesis-testing-via-out-of-sample-testing – Dave Sep 08 '20 at 19:48

Significance Test for Comparing RMSE values

0 Answers0