Why is sklearn PCA implementation in Python sensitive to the order of columns in source data?

Question

The rotation matrix outputted by the PCA algorithm should be independent of something trivial like the column ordering of the source data. Can anyone explain why my output diverges from my expectation for consistency?

I made a test input file 30x569 from a pre-made dataset

from sklearn.datasets import load_breast_cancer
cancer = load_breast_cancer()
cancer.keys()

df = pd.DataFrame(cancer['data'],columns=cancer['feature_names'])
df.to_csv(r'input file',index=False)

Then generated a 30x30 output with all the covariance-based PCA components

import pandas as pd
import numpy as np

daily_series = pd.read_csv (r'input path')

sd = daily_series[daily_series.columns[0:daily_series.shape[1]]]
scaled_data = sd #unscaled
from sklearn.decomposition import PCA
pca = PCA(n_components=daily_series.shape[1])
pca_model = pca.fit(scaled_data)
components = ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','PC8','PC9','PC10','PC11','PC12','PC13','PC14','PC15','PC16','PC17','PC18','PC19','PC20','PC21','PC22','PC23','PC24','PC25','PC26','PC27','PC28','PC29','PC30']
variables = daily_series.columns[0:daily_series.shape[1]]
Matrix = pd.DataFrame(pca_model.components_, columns=components, index=variables)

Matrix.to_csv(r'output path', index=True)

When I reorder the columns (let's say alphabetically) of the test input file. And run the above my output from the original test is different not just in the signs but also magnitude. I don't understand how that's possible.

Output (left is original output/right is output after alphabetizing columns in source data):

Answer 1

We have thoroughly developed the relationship between SVD and PCA in Relationship between SVD and PCA. How to use SVD to perform PCA? which is worth reviewing if you're uncertain about the connection.

---

The sklearn PCA implementation is working correctly.

The main thing to observe is that the SVD of $A$ is given by $$ A=U S V^\top $$ so for a permutation of columns via matrix $P$ we have $$ AP=U S V^\top P. $$

Another way to state this is that if you compute the SVD of $AP$, you'll end up with $AP = U S \tilde{V}^\top$, where $\tilde{V}^\top = V^\top P$.

We know that $\tilde{V}^\top=V^\top P$ is orthogonal because permutation matrices are orthogonal and products of orthogonal matrices are orthogonal.

Your screenshots show different things because you're comparing $V^\top$ and $V^\top P$, which are not equal in general. In fact, $V^\top$ and $V^\top P$ are only guaranteed to be equal if $P=I$. Column order matters just for $V$; $U$ and $S$ are the same.

---

We can even show that a permutation yields the same orthogonal rotation.

$$ \begin{aligned} AV &= USV^\top V \\ AV &= US \end{aligned} $$

And we can show the same result for $AP$ because a permutation matrix $P$ is orthogonal.

$$ \begin{aligned} AP P^\top V &= USV^\top P P^\top V \\ AV &= US \end{aligned} $$

In other words, the column order doesn't matter for creating a linearly independent basis for $A$, because you obtain the same result for $AP$ and $A$.

---

We can demonstrate this all in Python.

import numpy as np
from numpy.linalg import svd
from numpy.random import shuffle
from sklearn.datasets import load_breast_cancer

if __name__ == "__main__":
  X, y = load_breast_cancer(True)
  U, S, V = svd(X, full_matrices=False)

  P = np.eye(X.shape[1])
  shuffle(P)

  print("X and X @ P are not the same.")
  print(X @ P - X)

  # This will work correctly because both X and the SVD of X are permuted.
  assert np.allclose(U @ np.diag(S) @ V @ P - X @ P, 0.0)

  try:
    # This will fail because X is permuted but the SVD is ~not~.
    assert np.allclose(U @ np.diag(S) @ V - X @ P, 0.0)
  except AssertionError:
    print("V @ P != V")
    print(V @ P - V)

You can replace P with any permutation matrix you desire, even one which alphabetizes the column names.