I have some questions about the use of CV in machine learning, I created a post some days ago and I have been given a link to a previous post (very interesting). I also read this post (https://www.r-bloggers.com/2020/12/the-cross-validation-train-predict-misunderstanding/) about the misconception of cross validation and train-predict. If I understand correctly :
- Cross validation cannot allow me to choose a model over another
- Cross validation cannot allow me also to choose a set of hyperparameters among all the folds which seems very logic (bias induced+++)
So : I can only have an estimation of the average metric I chose on independent set with a very useful variance (the lower the better). I saw also some posts with explanations about the one standard error rule (Empirical justification for the one standard error rule when using cross-validation) but it confuses me...
In definitive, I can see with the CV if there is a stability between the inner and outer loop, and with the mean and sd of the performance (outer loop) I can choose a model performing better than others ? and in a case of equality, use the one standard error rule (for example) ?
I feel so stupid because there is 3 tons of posts about this and I still don't understand...
