1

I am trying to use Hugginface Datasets for speech recognition using transformers using this tutorial, epochs=30, steps=400, train_batch_size=16. Training loss, validation loss and WER decrease, and then increase:

[6321/7500 17:25:12 < 3:15:00, 0.10 it/s, Epoch 25.28/30]

Step  TrainingLoss ValidationLoss   Wer
400     4.171600    1.145224    0.914795
800     0.812200    0.489049    0.468949
1200    0.581000    0.625888    0.559847
1600    0.930700    1.078658    0.681997
2000    1.681100    2.083352    0.971417
2400    2.344900    2.128186    0.969882
2800    2.528900    2.261873    0.970472
3200    2.503300    2.261875    0.970472
3600    2.499400    2.261875    0.970472
4000    2.512800    2.261875    0.970472
4400    2.506000    2.261875    0.970472
4800    2.523700    2.261875    0.970472
5200    2.517800    2.261875    0.970472
5600    2.517600    2.261875    0.970472
6000    2.522000    2.261875    0.970472
....

Is this because I have too many epochs? Overfitting? Or does it have to do with steps/batch_size?

user1680859
  • 111
  • 3
  • Decrease your learning rate and increase model complexity but ensure that validation loss is not increased by this. Ensure you have representative training data. Early stop at the minimum validation loss. batch size also affects learning so you can set that as a hyperparameter. – Jose_Peeterson Oct 16 '22 at 17:16

0 Answers0