Using SVD to write the least squares fitted vector

Question

Elements of statistics p.66

Please I know the least squares solution for $\hat\beta = (X^TX)^{-1}X^Ty$ but I don't know how they were able to get

$X\hat\beta= X(X^TX)^{-1}X^Ty = UU^Ty$

These are the steps I followed :

$X\hat\beta = U\sum V^T(V\sum^T U^TU\sum V^T)^{-1}V\sum^T U^Ty \quad (1)$

$X\hat\beta = U\sum V^TV\sum^T U^Ty \quad(2)$

$X\hat\beta = UU^Ty \quad(3)$

Please from $(1)$ , why is $U^TU = I$ and $UU^Ty \neq I$?

I might have misrepresented the question in my answer. Are you asking why are $U$ columns orthogonal? — Firebug, Oct 27 '20 at 13:13

Firebug · Accepted Answer · 2020-10-27T14:10:48.297

3

From the the definition of SVD, $U$ and $V$ are orthogonal matrices. So the products $U^TU$, $V^TV$ are identity matrices.

\begin{cases} X = USV^T\\ X\hat\beta = X(X^TX)^{-1}X^Ty \end{cases}

So

$$X\hat\beta = (USV^T)((USV^T)^T(USV^T))^{-1}(USV^T)^Ty\\ =USV^T(VS\color{red}{U^TU}SV^T)^{-1}VSU^Ty\\ =USV^T(VS^2V^T)^{-1}VSU^Ty\\ =US\color{red}{V^TV}S^{-2}\color{red}{V^TV}SU^Ty\\ =U\color{red}{SS^{-2}S}U^Ty\\ =UU^Ty\\$$

On why $UU^T$ is not necessarily $\mathbb I$, this stems from the definition of the SVD. A compelling argument can be given by the last line of our derivation, though: $X\hat\beta$ is not necessarily equal to $y$, thus $UU^T$ is not necessarily the identity matrix.

edited Oct 27 '20 at 14:10

answered Oct 27 '20 at 13:09

Firebug

19,076
6
77
139

1

from your steps (3) why is $ (VS^2V^T)^{-1} = VS^{-2}V^T $ but not $V^{-1}S^{-2}V^{-T}$ – EA Lehn Oct 27 '20 at 13:17
@EALehn that's a property of $V$, $V^T=V^{-1}$. The SVD is often used to calculate matrix powers, like $X^n = US^nV^T$ – Firebug Oct 27 '20 at 13:56
so $(VS^2V^T)^{-1} = V^TS^{-1}V$ is also correct or – EA Lehn Oct 27 '20 at 14:08
@EALehn No, it's not: $(VS^2V^T)^{-1} = (V^{-T}S^{-2}V^{-1}) = VS^{-2}V^T$ – Firebug Oct 27 '20 at 15:01

Using SVD to write the least squares fitted vector

1 Answers1

Linked