Relations between regularization constant and parameter space in ridge regularization

Question

Can it be shown that the $||w^*||_2$, where $w^* = (X'X+\lambda I)^{-1}X'Y$ , is inversely affected by the regularization constant $\lambda$, i.e. it is of $O(\frac{1}{\lambda})$ ?

Answer 1

Let the SVD decomposition of X be $$X = UDV^T,$$

where U and V are orthonormal matrices, and D is a diagonal matrix and the diagonal is $(d_1, d_2, \ldots, d_p)$

Then

$w^*(\lambda) = (VD^2V^T + \lambda I)^{-1}VDU^TY$

$\ \quad = (VD^2V^T + \lambda VIV^T)^{-1}VDU^TY$

$\ \quad = V(D^2 + \lambda I)^{-1}V^TVDU^TY$

$\ \quad = V(D^2 + \lambda I)^{-1}DU^TY$

$\|w^*(\lambda)\| = \|(D^2 + \lambda I)^{-1}DU^TY\| \quad \quad$ (since multiplying an orthonormal matrix does not change norm)

$\ \quad = \sqrt{\sum_{i=1}^p R_i^2(\lambda) (u_i^TY)^2}$

where $R_i(\lambda) = \frac{d_i}{d_i^2 + \lambda}$ and $u_i$ is the ith column of $U$.

The values of $d_i$'s or $u_i^TY$'s remain constant as you change $\lambda$. You can see the whole expression is on the $O(\frac{1}{\lambda})$ scale as $\lambda$ goes to infinity.