While fine-tuning a deep neural network I ran into the following situation:
- My train- and validation loss are both decreasing and have very similar values throughout training. Especially the train-loss is not significantly lower than the validation loss. Still both loss values are rather high. Thus, I would argue the model is underfitting.
- After training is complete, I am calculating various metrics such as precision and recall on the trainingset and the validationset. Here, the metrics on the trainingset seem sound but the metrics on the validationset show very poor performance. Thus, I would argue the model is overfitting.
This seems very contradictory to me. Usually I would argue that the loss function I am using is not useful for the task I want the model to learn, however, I do not have the possibility to change it.
What can I do now? And: is my model underfitting or overfitting?