8

Let $A$ a be an $n$ x $p$ matrix and let $B$ be the transformed data set of $A$ under $Q$:

$$ B = A Q $$

where Q is a $p$ x $p$ orthogonal matrix:

$$ Q Q^T = I $$

$n$ is the number of samples (rows) and $p$ is the number of features (columns).

 

$A$ and $B$ are centred (zero mean for each feature).

Now suppose we perform PCA on both $A$ and $B$:

$$ \frac{1}{n-1} A^T A = V_A L_A V^T_A $$

$$ \frac{1}{n-1} B^T B = V_B L_B V^T_B $$

The question is, are the projections of $A$ and $B$ on their principal directions equal and why?

$$ A V_A = B V_B \ ? $$

That is, are the datasets identical upon "transforming" them with PCA?

Later edit: I simplified the question to only refer to orthogonal transformations (rotations). The original question referred to translations as well (the explanation in that case is trivial).

turdus-merula
  • 1,611
  • 19
  • 22
  • 1
    You may want to change "Suppose $A$ and $B$ are two $n x p$ matrices representing two data sets" to something along the lines: "let $A$ a be an $n x p$ matrices and let $B$ be the transformed data set of $A$ under $Q$." just for clarity – Joseph Santarcangelo Oct 15 '16 at 16:45
  • @JosephSantarcangelo thanks, I've fixed that. Feel free to adjust the question if it isn't clear enough. – turdus-merula Oct 15 '16 at 17:02

2 Answers2

5

The covariance matrix of $B$ is decomposed as:

$$ \frac{1}{n-1} B^T B = V_B L_B V^T_B $$

If we rewrite the covariance matrix of $B$ in terms of $A$ and $Q$ we have:

$$ \frac{1}{n-1} B^T B = \frac{1}{n-1} (AQ)^T AQ = \frac{1}{n-1} Q^T A^T A Q = Q^T (\frac{1}{n-1} A^T A) Q = Q^T V_A L_A V^T_A Q $$

We observe that the covariance matrices of $A$ and $B$ are generally different:

$$ V_A L_A V^T_A \neq Q^T V_A L_A V^T_A Q $$

We identify (wasn't sure about this step at first):

$$ V_B = Q^T V_A $$ $$ L_B = L_A $$ $$ V^T_B = (Q^T V_A)^T = V^T_A Q $$

And now we have:

$$ B V_B = A Q Q^T V_A = A V_A $$

Which means that the datasets are identical upon "transforming" them with PCA.

Many thanks to @amoeba and @Alex R.

turdus-merula
  • 1,611
  • 19
  • 22
  • 1
    +1, this is correct. The step you are not sure about is legit because the eigen-decomposition is unique (assuming that all eigenvalues are distinct). So if you know that $Q^\top V_A$ is an orthogonal matrix and that $L_B$ is a diagonal matrix, then you know that $Q^\top V_AL_AV^\top_AQ$ is the eigen-decomposition. – amoeba Oct 15 '16 at 21:23
1

Let $R$ be a rotation (i.e. an orthogonal matrix: $R^T=R^{-1}$). $A=U\Sigma V^*$ by SVD, and correspondingly $A^TA=V\Sigma^2 V^*$.

On the other hand if $B:=RA$, then:

$$B^TB=A^TR^TRA=A^TA=V\Sigma^2V^*.$$

So PCA is "invariant" under rotations applied to $A$ from the left. Specifically, this means that the covariance matrix is invariant. However the actual projections will differ. Specifically, $A=U\Sigma V^*$, whereas $B=RA=RU\Sigma V^*$, giving $B=W\Sigma V^*$, where $W:=R^*U$.

Alex R.
  • 13,897
  • I will "accept this answer" in a few days. I would like to clean up my question a bit first. – turdus-merula Oct 10 '16 at 08:04
  • 2
    @user115202 I don't think this answer is correct. "Rotating" a data cloud will change the covariance matrix (the eigenvectors will rotate too), but the projections will stay the same. Because such a rotation corresponds to rotations applied to $A$ from the right, not from the left, assuming $A$ is $n\times p$. – amoeba Oct 15 '16 at 00:16
  • @amoeba and Alex, I attempted an explanation. Do you think you can check it sometimes? Thank you for your answers and comments. – turdus-merula Oct 15 '16 at 17:37
  • -1. I am sorry, I feel I have to downvote this answer because in its current form it can be misleading. Please ping me if you edit, Alex, I will be happy to remove the downvote. CC to @user115202. – amoeba Oct 15 '16 at 21:22