How to prove the LOOCV formula for smoothing matrix?

Question

When I read The Elements of statistical learning, in the section of "5.5 Automatic Selection of the Smoothing Parameters", equation (5.26) gives a LOOCV formula for corresbonding $\lambda$

Here the background is fitting a smoothing spline using the data $(y_i, x_i)\ i = 1,2,...,N$. In order to select a smoothing parameter, we use LOOCV creteria.

\begin{equation} \text{CV}(\hat{f}_{\lambda}) = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{f}_{\lambda}^{(-i)}(x_i))^2=\frac{1}{N}\sum_{i=1}^{N}\left(\frac{y_i - \hat{f}_{\lambda}(x_i)}{1-S_{\lambda}(i,i)}\right) \end{equation}

where $\hat{f}_{\lambda}^{(-i)}$ means fitting value by using model without using $(y_i, x_i)$, and $S_{\lambda}(i,i)$ is the diagonal element of the smoothing matrix $\mathbf{S}_{\lambda}$ based on whole data. ($\mathbf{S}_{\lambda} = (\mathbf{I} + \lambda \mathbf{K})^{-1} = \mathbf{N}(\mathbf{N}^T\mathbf{N} + \lambda \Omega)^{-1} \mathbf{N}^T)$.

(the notation is consistent with chapter 5 in the textbook The Elements of statistical learning)

I wonder how to prove the second equation, so I search online and find the following post is helpful. But when I try to prove the above formula for smoothing matrix, I've run into difficulties and it's hard to carry on.

I can only prove that

\begin{equation} (\mathbf{N}_{(t)}^T\mathbf{N}_{(t)} + \lambda \Omega)^{-1} \mathbf{N}_t^T = (\mathbf{N}^T\mathbf{N} + \lambda \Omega)^{-1} \mathbf{N}_t^T \left(\frac{1}{1-S_{\lambda}(t,t)}\right) \end{equation}

where the supscript $_{(t)}$ means except $t$th one, while $_t$ means the $t$th one.

Could anyone help finish the proof, or using some other method to prove the formula? I'd be grateful.

Helpful post:

@MISC {164223, TITLE = {Proof of LOOCV formula}, AUTHOR = {Clarinetist (https://stats.stackexchange.com/users/46427/clarinetist)}, HOWPUBLISHED = {Cross Validated}, NOTE = {URL:Proof of LOOCV formula (version: 2015-08-04)}, EPRINT = {https://stats.stackexchange.com/q/164223}, URL = {https://stats.stackexchange.com/q/164223} }

Answer 1

I do not know if you are still interested in this but what follows might help you in proving the results you mention in the question (...hope get everything right...it is a slippery notation ^_^).

Here I will use a bit more general notation (since the results you mention hold not only for the smoothing use case)

• We can consider a ridge regression problem $\displaystyle \min_{\beta} S(\lambda) = \| y - X \beta \|^{2} + \lambda \|\beta\|^{2}$ (if $\lambda = 0$ you will get the same results as the answer you cite).
• The LOOCV ctiretion in this case tells us to select the optimal regularization parameter as $LOOCV(\lambda) = \displaystyle \min_{\lambda} \sum_{i}^{N}(y_{i} - X_{i} \hat{\beta}_{-i} (\lambda))^{2}$
• We define $X_{i}$ as the $i$th row of the model matrix $X$
• We define as $\hat{\beta}_{-i} = (X_{-i}^{\top} X_{-i} + \lambda I)^{-1} X_{-i}^{T} y_{-i}$ the coefficients estimated when the $i$th obs is left out
• Define as $X_{-i}$ the the model matrix when the $i$th element is left out (analogous definiiton holds for $y_{-i}$)

To derive the result you are interested in we need to find an 'analytic' expression for $\hat{\beta}_{-i}$. The proof proceeds more or less as follows (I will skip some details):

• The inverse appearing in $\hat{\beta}_{-i}$ can be shown to be equal to (this is an application of the Woodbury identity) $$ (X_{-i}^{\top} X_{-i} + \lambda I)^{-1} = (X^{\top} X + \lambda I)^{-1} + (X^{\top} X + \lambda I)^{-1} X_{i}^{\top} [1-H_{ii}(\lambda)]^{-1} X_{i}(X^{\top} X + \lambda I)^{-1} $$ where $H_{ii}(\lambda) = X_{i}(X^{\top} X + \lambda I)^{-1} X_{i}^{\top}$

• Plugging the result above in the equation for he LOO estimator, after some algebra, we get: $$ \hat{\beta}_{-i}(\lambda) = \hat{\beta}(\lambda) - (X^{\top} X + \lambda I)^{-1} X_{i}^{\top} [1-H_{ii}(\lambda)]^{-1} (y_{i} - X_{i} \hat{\beta}(\lambda)) $$

• Plugging the result in the equation for the LOOCV criterion, after some algebra, we can notice that $$ y_{i} - X_{i} \hat{\beta}_{-i}(\lambda) = [1-H_{ii}(\lambda)]^{-1} (y_{i} - X_{i}\hat{\beta}(\lambda)) $$ and hence the results we are looking for follows: $$ LOOCV(\lambda) = \sum_{i=1}^{N} \frac{(y_{i} - X_{i}\hat{\beta}(\lambda))^{2}}{(1 - H_{ii} (\lambda))^{2}} $$

Again, here I skip some detail (mainly tedious algebra) but these are the lines you could follow to prove the results you are interested in. I hope this helps.