0

Most learning (training and validation) curves that I see in literature and through online resources depict a steep decrease/increase in loss/accuracy during initial epochs followed by a subsequent flattening/plataeuing of these curves. As the performance of a model is generally measured by its accuracy on the validation set, what is the best approach to take when one is not satisfied with the level of validation accuracy obtained with the current data and model used?

Assume that the model is sufficiently regularized and that this level of accuracy is obtained for both overfitting and (slightly) underfitting (enough that the model is still capable of learning, i.e., the training loss continues to decrease) regimes using the same architecture. Is the best solution (as I have gathered online) to simply gather more data until one achieves a desired level of (generalization) accuracy?

kjetil b halvorsen
  • 77,844
Messiah
  • 1
  • 1
    More data does not guarantee more accuracy. For instance, if you're trying to predict the height of a wife from that of her husband, you're still going to have a relatively large error even when you have a complete census of the population and every socioeconomic status variable ever measured. https://stats.stackexchange.com/questions/352036/what-should-i-do-when-my-neural-network-doesnt-learn/352037#352037 might be relevant. – whuber Jun 01 '22 at 21:48
  • @whuber Thanks for your response. Let's say the the features I am feeding into the model are highly relevant to the output, and that an arbitrary number of datapoints can be generated to train this model. I observe that validation/generalization accuracy improves with the addition of more data. Obviously this makes sense, since the model is better able to understand the data distribution from which the samples are drawn, however, is this indeed the most significant source of improving the model? I have already incorporated the points mentioned in the link you provided. – Messiah Jun 01 '22 at 22:04
  • It depends on the circumstance. Everything in your comment still conforms with the example I gave: there's no guarantee in general that one variable is deterministically related to any number of other variables: there will be a certain amount of "irreducible error," as it's sometimes called. – whuber Jun 02 '22 at 13:09

0 Answers0