1

When using MSE (or RMSE) loss for regression tasks in deep learning my models usually output the same result for each record. My guess is that the models are "playing safe" and they output a value towards the mean instead of learning anything.

Does anyone have encountered this problem or knows how to solve it?

Alejo Paullier
  • 11
  • I believe that this is not a loss issue. However, I think you need to provide more info about your task. A reasonable presumption is that your data does not correlate with your goals. – Allonsy Jia Oct 04 '22 at 01:35

2 Answers2

1

The (R)MSE elicits the conditional mean or expectation of the unknown distribution of your observables. If there is no structure in your data, then the best guess is that they are IID, so their expectation is identical, and the RMSE-optimal prediction of each instance is identical. (There is nothing special about the RMSE here - if your observables look IID, then any loss function will yield the same prediction.) This behavior is not surprising.

If you find it surprising, then you presumably have an idea about structure in your data that your model is not picking up. Great! Help your model understand about this structure!

You may find this helpful: How to know that your machine learning problem is hopeless?

Stephan Kolassa
  • 123,354
0

My first thought was that this indicates the variables being unrelated to the outcome, in agreement with Stephan Kolassa.

However, if a large neural network cannot even overfit, that suggests to me some issue with the optimization. If your variables are unrelated to the outcome, you should expect poor out-of-sample performance, but you should be able to get something out of the in-sample data.

Dave
  • 62,186