2

Suppose we want to tune the hyperparameters of an algorithm. We perform $k$-fold cross validation and we found the optimal hyperparameter values, lets say $p^*$. Nevertheless, we don't have a separate test. Can we report the average error from folds as an estimate of the generalization error?

Based on this Question we shouldn't.What I can't understand is the following.

Lets say that another analyst wants to estimate the generalization performance of $p^*$. He performs $k$-fold cv with the same data we used for hyperparameter tuning. He doesn't want to tune parameters. Just by chance he selected $p^*$ and wants to get an estimate of its performance. Further, lets assume that his $k$-folds are same to our $k$-folds. Obviously the average cv error would be the same. Why he can report this error as an estimate of the generalization error while we are not allowed?

ado sar
  • 471
  • 3
  • 11
  • 1
    Because you used the validation folds data to choose hyperparameters which minimise the apparent errors, and part of this process may have fitted noise in a way which biases the results downwards – Henry Nov 08 '22 at 12:24
  • 1
    @Henry Can't the same being said for the analyst who just wants to estimate performance? – ado sar Nov 08 '22 at 12:41
  • That depends on how and why that analyst chose those hyperparameters. If they did it because of a different study using different data suggested this then they are fine as the new data is in effect a test set. If they did it because you suggested the model and hyperparameters using the same data as they are planning to use, then they are in effect you and you will get the same bias. – Henry Nov 08 '22 at 12:57

0 Answers0