1

In a Neural Network training, the cost of the model changes throughout the training process when using gradient descent (or something analogous), this is the point of the algorithm. However the cost might not decrease monotonically. In some points the cost might even increase.

Cost in Neural Network model per 100 iterations using gradient descent

So, Is it OK to keep track of the parameters that output the lowest cost and use those as the best parameters for the model? In the image it would imply using the parameters that output the lowest cost instead of the last parameters found.

Assume that the cost returned by the lowest cost outputs an acceptable model accuracy.

Does doing this cause some kind of problem?

loco.loop
  • 111

1 Answers1

0

This is a perfectly okay thing to do. Other things you might try are reducing the learning rate to make it more likely that the training cost goes down most of the time and also selecting the model based on the cost on the validation set rather than training.

Aaron
  • 3,275