1

At this point in a talk by Nando de Freitas, there is an answer to an audience question, about how theory has got left behind in statistics, but theory is still important, and he gives an example where there was an observed phenomenon in machine learning, and the theorists went away and confirmed it exists... and he seems to be saying that it is: at the point where you see the validation/test data set error start to rise (where we always use early-stopping to prevent over-fitting), if you just carry on eventually you reach another turning point and the error comes back down again.

I've never seen this second turning point, even when deliberately over-training to see what happens. Instead what I see is (if the model has enough capacity) it perfectly memorizes the training data, and the validation error levels out (at something bad).

Does anyone know what phenomenon he is describing, or what my misunderstanding of what he is saying is? (Links to paper(s) on this would also be welcome.)

Darren Cook
  • 2,107
  • 1
  • 14
  • 26
  • What do you mean by "carry on"? If you have an infinite amount of representative training data, you will eventually have a "perfectly-trained model". Overfitting is often a consequence of a very little (or not representative) sample – David Jun 13 '19 at 07:47
  • @David I was thinking in terms of deep learning (where "carry on" means another epoch on your training data), or GBM/random-forest (where "carry on" means adding another tree, using the same training data). – Darren Cook Jun 13 '19 at 08:26
  • Then... No. Not at all. You need to continue the training with fresh data – David Jun 13 '19 at 11:36

0 Answers0