1

My question is very simple, but no matter where I look it up, it seems that I get another answer.

Take a simple classification task. Let's say I trained a kNN, LDA and logistic regression on it for simplicity. For kNN I already determined the optimal number of neighbors (hyperparameter) via cross validation.

Then what do? Considering the book I am currently reading, I should only touch the test data once. So, would I run each model on the test data once and just pick the one, which does best on it? I mean chances are big that one model does better than the other by accident? So I would be tempted to repeat the processs and taking some kind of average (just like picking the hyperparamter by crossvalidation). But since that would be contradicting to the what I read so far.

I did some research on the internet and also watched some videos on this topic. Bur it seems like every other person has another opinion.

Can you describe what is the best to be done or point me to some state of the art resource?

Marlon Brando
  • 435
  • I have marked this as a duplicate. The accepted answer over there can be summarized using just its first sentence: You could consider the model itself a hyperparameter as well. – Dave Mar 14 '24 at 19:07
  • Thanks for the reference, Dave. So if I understand correctly, my book is right in the sense that test data should never be touched more than once and this paradigm even includes different models! So after reading through the answers, I would need to split the test right at the beginning, before even starting with cross-validation, because otherwise different models would use different data as test sets, which would violate the unseen data principle at the end of the process (after choosing the model which performed best on the validation set). Is that how you would approach the problem? – Marlon Brando Mar 14 '24 at 20:28

0 Answers0