SVD in linear regression

Question

I was reading the book Elements of Statistical Learning and came across the section that tried to interpret ridge regression using singular value decomposition (SVD) of the design matrix, $X$. Specifically, I found the following:

$X=UDV^{T}$, where matrix $U$ is $N\times p$, $V$ is a $p\times p$ orthogonal matrix, and $D$ is a $p\times p$ diagonal matrix.

I am confused because from Wikipedia, the orthogonal matrix has to be a square matrix. In this case matrix $U$ does not qualify. Later I tend to believe that $U$ contains orthogonal columns only, and that results in $U^{T}U=I$, but $UU^{T}\ne I$. This seems to make sense because I found in the book

$X \hat{\beta}=X(X^{T}X)^{-1}X^{T}Y=UU^{T}Y$, and $UU^{T}Y$ should not be equal to $Y$

So my question becomes: are there two versions of SVD I can do? One results in both $U$ and $V$ being orthogonal and square matrix, and the other like this? Or is there anything wrong with my argument?

Any guidance is appreciated.

Update after receiving initial answer:

After reading @BabakP 's answer, I thought testing the algorithm using software is a good idea. So I tried svd function in Matlab. The result shows a square U matrix in dimension NxN, a diagonal matrix D in dimension Nxp, and a square V matrix in dimension pxp. Example below:

A=[ones(10,1) randn(10,1)];
[U,S,V]=svd(A);
>> size(U)

ans =

10    10

>> size(S)

ans =

10     2

>> size(V)

ans =

 2     2 

So does this mean R and Matlab give two different versions?

Answer 1

As far as I know there is only one version of SVD. The correct dimensions for an SVD decomposition are $N\times p_1$, $p_1\times p_2$ and $M\times p_2$, this makes sense because you want the product of the three matrices to be (a reconstruction of) the original matrix. So if $X$ is $N\times M$, so should the reconstruction be, or to put it differently:

$$N \times M = (N \times p_1) \times (p_1 \times p_2) \times (M \times p_2)^T$$

Edit: usually, $p_1 = p_2 = p$, resulting in a square matrix (like in Matlab)

The orthogonal, rectangular matrices contain left and right singular vectors respectively and the middle, rectangular matrix contains the singular values on the diagonal.

Edit2: (see comments)

A=[ones(10,1) randn(10,20)];

[U,S,V] = svd(A);
errors = zeros(10,1);
for p = 10:-1:1
    err = U(:,1:p) * S(1:p,1:p) * V(:,1:p)' - A;
    errors(p) = sum(sum(err.*err));
end
plot(errors);
ylabel('Squared error');
xlabel('p');

Answer 2

Here is some R code that validates your formulas given above:

#Generate psuedo data
Y = rnorm(10)
X = matrix(c(rep(1,10),rnorm(10)),ncol=2)

#Calculate X times beta hat
XB1 = X%*%solve(t(X)%*%X)%*%t(X)%*%Y

#Make sure X = UDV'
svd(X)$u%*%diag(svd(X)$d)%*%t(svd(X)$v)

> svd(X)$u%*%diag(svd(X)$d)%*%t(svd(X)$v)
      [,1]        [,2]
 [1,]    1 -0.20283033
 [2,]    1 -0.85846798
 [3,]    1  0.07970559
 [4,]    1 -0.28254373
 [5,]    1  0.39261439
 [6,]    1 -0.31559482
 [7,]    1  0.20561526
 [8,]    1  0.55152336
 [9,]    1 -0.69396930
[10,]    1 -1.21970880
> X
      [,1]        [,2]
 [1,]    1 -0.20283033
 [2,]    1 -0.85846798
 [3,]    1  0.07970559
 [4,]    1 -0.28254373
 [5,]    1  0.39261439
 [6,]    1 -0.31559482
 [7,]    1  0.20561526
 [8,]    1  0.55152336
 [9,]    1 -0.69396930
[10,]    1 -1.21970880


#Calculate UU'Y
U = svd(X)$u
XB2 = U%*%t(U)%*%Y

#Check to see if they return the same thing
cbind(XB1,XB2)

> cbind(XB1,XB2)
            [,1]       [,2]
 [1,] -0.4644321 -0.4644321
 [2,] -0.7215807 -0.7215807
 [3,] -0.3536183 -0.3536183
 [4,] -0.4956966 -0.4956966
 [5,] -0.2308919 -0.2308919
 [6,] -0.5086596 -0.5086596
 [7,] -0.3042351 -0.3042351
 [8,] -0.1685660 -0.1685660
 [9,] -0.6570624 -0.6570624
[10,] -0.8632634 -0.8632634

So as you can see from the output above, for sure one decomposition of $X$ is $X=UDV^T$. Likewise, calculating $UU^TY$ is equivalent to calculating $X\hat\beta$ where $\hat\beta=(X^TX)^{-1}X^TY$. So this solution really just pertains to validating your second question about whether or not what you are doing is correct.