7

Consider a set $X=(X_1; \dots; X_n)$ of $n$ data points such that $X_i \in \mathbb{R}^d$ is a column vector. Let $Y = \text{pca_proj}(X)$ denote the projection of points in $X$ according to the PCA components i.e. $$Y_i \in \mathbb{R}^d\\ Y_i = W X_i$$ where $W \in \mathbb{R}^{d \times d}$ is the projection matrix obtained from PCA, and each row of $W$ is an eigen vector of $X^T X$.

Now let's define $Z = \text{pca_proj}(Y)$. My questions is: does $Y = Z$? If not, what happens if we keep applying PCA recursively on a matrix? Does it converge to a specific matrix?

Sobi
  • 2,251
  • 1
    Since PCA can be (and usually is) defined without reference to any particular basis of $\mathbb{R}^d$, the answer is immediate. If that's not perfectly clear, then you might enjoy reading over many of the answers at http://stats.stackexchange.com/questions/2691 . – whuber Mar 10 '16 at 22:22
  • 1
    Subjecting PCs of PCA again to PCA isn't helpful because PCs are orthogonal (uncorrelated) variables: the covariance matrix is diagonal. – ttnphns Mar 10 '16 at 22:30

1 Answers1

7

PCA at it's heart involves diagonalizing a matrix which means solving for the eigenvalues and eigenvectors of said matrix. The whole purpose of the calculation is to find a diagonal representation of your matrix (i.e. only elements along the diagonal of the matrix). If you solve again, you will find that you are trying to calculate the eigenvalues of a diagonal matrix, which yields the exact same diagonal matrix. Hence your PC vectors will be the same regardless of how many times you apply the transformation.

Greg Petersen
  • 361
  • 1
  • 6
  • 2
    The conclusion relies on the implicit assumption that all eigenvalues are distinct. When they are not, and the PCA algorithm chooses a random basis for each eigenspace, then your conclusion will be incorrect. This actually happens in R. – whuber Mar 11 '16 at 15:40
  • 1
    Yes this is true. However, I would say that this is a natural ambiguity that results from having degenerate values and is not specific to the case of multiple applications of PCA analysis. Essentially if you have the ambiguity on your first iteration, you will still have it on your second iteration. Thus, nothing has changed from applying PCA twice. Also, the analysis and physics will be the same regardless of the arbitrary basis that was selected by any software package. – Greg Petersen Mar 11 '16 at 19:14