Question about retraining a regression model

Question

Exercise. Suppose you train a Ridge model to a regression problem that has a normalized perfomance measure (say K) that attains a value in the interval [0,1], where 0 means that the model is terrible and 1 means that the model is perfect. Suppose that you evaluate the trained model and obtain in the training set K = 0.22 and in the test set K = 0.21. Which of the following options should you apply?

$ (a) $ Train the model again decreasing the value of the regularization coefficient, $\lambda$ ;

$ (b) $ There isn't need to do anything. The model obtained is satisfatory ;

$ (c) $ Train the model again increasing the value of the regularization coefficient, $\lambda$.

My attempt. According to the values of $K$, our adjust to the data is clearly not satisfatory, in the sense that the values for $K$ are pretty small. Since, for higher values of $\lambda$, we know that the adjust to data is worse and the complexity of the model is more important, if we want to obtain better values for $K$, we should train our model again, with a smaller value for our regularization coefficient, $\lambda$, to give us better results for $K$. Therefore, I would choose option $(a)$.

What ChatGPT says. Given that the performance measure K is normalized and attains a value between 0 and 1, we can conclude that the model's performance is not terrible but not perfect either. The difference in the performance measure between the training and test sets is also not significant. Therefore, there is no need to retrain the model with a different value of lambda. Option (b) is the correct choice: there is no need to do anything, and the model obtained is satisfactory.

My question. I agree that the difference between training and test sets is not significant but I am not so sure about the perfomance measure is good at all. Is ChatGPT wrong here?

Thanks for any help in advance.

Answer 1

My take: neither you, nor ChatGPT, nor a-c are correct.

You write:

According to the values of $K$, our adjust to the data is clearly not satisfatory, in the sense that the values for $K$ are pretty small.

That logic is incorrect. Suppose your task is to predict the flip of a fair coin, and $K$ is given by the accuracy of your prediction over many trials, but squared. (We disregard the fact that accuracy is a horrible KPI.) Since accuracy lies between 0 and 1, squared accuracy also lies between 0 and 1, so $K$ is as in your question. However, since this is a fair coin, our accuracy cannot be expected to be above 0.5, so $K$ cannot reasonably be expected to exceed $0.5^2=0.25$. So just because we got a "small" value for $K$ does not mean it is "not satisfactory", or that it can be improved.

Thus, the pure fact that $K$ has a certain value does not tell you anything. Rather, what we should be doing (and this is IMO the correct answer) is to assess in what circumstances our model does a bad job, and try to address these situations. See How to know that your machine learning problem is hopeless?

What the question or your professor probably wanted you to argue is: "Since test and validation loss are similar, there is no clear evidence of overfitting. Therefore, we can safely retrain the model with less regularization, i.e., lower values of $\lambda$." Yes, that reasoning makes sense... but I find my take above more helpful, if only because your software (like GLMNet) will probably optimize the regularization parameters automatically, so thinking about a particular value of $\lambda$ is usually not very useful.