0

I happened to an interview question:

In Ridge regression, what does it imply if the out-of-sample performance never change however we tune the hyperparameter (the coefficient of L2 regularization)?

I only know that there exists an optimal coefficient which makes the MSE of ridge smaller than its OLS estimation. I am not sure if it is related to this question and I have the no idea for the answer.

kjetil b halvorsen
  • 77,844
user6703592
  • 1,325
  • What is L2 regularization intended to accomplish? Think, then, of what it can accomplish when that objective is unnecessary: namely, when all the explanatory variables already are orthogonal. – whuber Jul 30 '21 at 20:00
  • @whuber does that mean when the multicollinearity is not heavy, change of $\lambda$ cannot improve the performance (reduce MSE) significantly? Here mentioned the out-of-sample, why do we specially use this? – user6703592 Jul 30 '21 at 20:30
  • That is the heart of the question. It makes one wonder exactly what "out-of-sample performance" really means. Exactly what observations constitute "out of sample"? Would it be referring to cross-validation? Testing on a hold-out set? Testing on new data altogether? Once you have decided that, it would be possible to analyze the situation mathematically. I find the approach described at https://stats.stackexchange.com/a/164546/919 to be particularly helpful at visualizing what might be going on. – whuber Jul 30 '21 at 20:48
  • @whuber many thanks. And could you also refer me some answers of "why ridge can reduce the multicollinearity"? Since up to now, I can only explain: multicollinearity increases estimation variance -> ridge reduces variance -> therefore ridge reduces multicollinearity. But actually this logic does not convince me strongly, It is better to have some detailed deduction. – user6703592 Jul 31 '21 at 06:06
  • @whuber very nice explanation of extended $n+p$ matrix. But i still don't understand "out-of-sample performance" here, could you open an answer here to discuss more details? Since recently I happened too many questions of ridge and multicollinearity. – user6703592 Jul 31 '21 at 07:05

0 Answers0