2

One of properties of PCA states that the sum of the variances of the principal components is equal to the sum of the variances of the explanatory variables. I wonder how to interpret this as I've always thought that we do not consider $X$'s as random variables. I'm quite new to probability theory and I need to get it straight: are explanatory variables random variables (or do we consider them fixed)? And if we don't consider them random, how is it possible to apply the variance operator to a variable that is not stochastic?

gung - Reinstate Monica
  • 145,122
Pasato
  • 331
  • The claim in your first sentence—"One of properties of PCA states that sum of the variances of the principal components is equal to the sum of the variances of the explanatory variables."—is only true when performing PCA on the covariance matrix $\mathbf{\Sigma}$. However, when performing PCA on the correlation matrix $\mathbf{R}$, the assumption is that each variable contributes precisely 1 unit of variance to total variance. – Alexis Jun 10 '14 at 22:05
  • I hope it wasn't too misleading. PCA was just an example to display the problem I had trouble with :) – Pasato Jun 10 '14 at 22:38
  • Well it is misleading in that it is often false. – Alexis Jun 10 '14 at 22:46
  • 3
    PCA is simply an applied mathematical method. It itself doesn't assume that the variables being analyzed are random. PCA is indifferent to words "sample", "population" etc. – ttnphns Jun 11 '14 at 05:38
  • 2
    @Alexis, saying that it is false is too strong. When the correlation matrix is being analyzed, in essence the variables have all been standardized. Thus the variables all have variances equal to 1 & "the sum of the variances of the principal components is [still] equal to the sum of the variances of the explanatory variables". The question is still sensible even in the case of the correlation matrix. – gung - Reinstate Monica Jun 11 '14 at 17:20
  • @gung, I am afraid I disagree: PCA of the covariance matrix explicitly links proportion of variance to the actual magnitude of each variable's variance. PCA of the correlation matrix does not actually transform (standardize) any of the actual variables, which actually may have incomparable variances. – Alexis Jun 11 '14 at 18:05
  • @Alexis, please don't be diehard. I'm sooner with gung, not with you. PCA on correlations is PCA of standardized variables. The problem with your stand is that actual magnitude for variance. The thing is that when you do PCA on correlations, the "actual magnitude" for you is 1. Don't like it? Then don't do PCA on corrrelations. – ttnphns Jun 11 '14 at 19:07
  • @ttnphns The problem is that $\text{cor}(\mathbf{X}) = \text{cor}(a\mathbf{X}+b)$ for any real $0 < a < \infty$ and any $-\infty < b < \infty$. So, no: PCA on correlation matrices does not require standardized variables, rather it requires an interpretation of all variables contributing equally to the PCA concept of "total variance." I get where you are coming from. I just don't agree with you. XO, Senhora Diehard :D – Alexis Jun 11 '14 at 19:12
  • @Alexis, don't pick on words, please. Interpretation of all variables contributing equally to the PCA concept of "total variance." is ornate and somewhat obscured wording of "variables are taken as having equal variance" (1 or another magnitude - no difference). – ttnphns Jun 11 '14 at 19:27
  • @ttnphns I thought I was picking on the math? The results of PCA on the correlation matrix are the same whether or not one standardizes ones variables (see my previous comment for why). If PCA of the correlation matrix required standardization, then that mathematical fact would not hold. Ergo, the OP's first sentence is mathematically false. – Alexis Jun 11 '14 at 19:39
  • 1
    @Alexis, maths operations imply something beyond what they show. (Just like words. If it were not so, no philosophy could be ever build on both.) Computing correlations imply standardizing variables, even though you don't calculate those. – ttnphns Jun 11 '14 at 19:48

1 Answers1

3

A variable is anything that does (or even can) take different values. Not all variables are random variables. When we do regression analyses, we consider the explanatory / predictor variables to be fixed, even when we sampled their values. This is because we are interested in understanding the response as a function of the explanatory variables. In another context, and with different goals, we can take the explanatory variables as stochatic, if appropriate. There is an interesting philosophical issue here, but it is a bit moot. Even in laboratory experiments, where all variables are controlled and set a-priori exactly at fixed levels, the explanatory variables certainly do have variances (albeit PCA would be tremendously uninteresting in such a case).

gung - Reinstate Monica
  • 145,122
  • Thanks gung. By the way - is it mathematically legit to take variance of variable which we do not assume to be random? – Pasato Jun 10 '14 at 21:48
  • 1
    Variance is simply a property of a set of numbers. The existence of a variance does not presuppose that the variable was stochastic. You can take a variance of a variable fixed at prespecified levels. What you might legitimately conclude from the variance may be another issue, though. – gung - Reinstate Monica Jun 10 '14 at 21:51