Factor analysis and Dimensional Reduction difference

Question

In exploratory factor analysis, we can clearly see the weightage of the factors on different features. Following this, we can understand the extent to which factors have an influence on the features. I understand this as factor analysis.

In other techniques, be it supervised like PCA or unsupervised like LDA, is the aforesaid possible? For example in PCA, the factors get boiled down to an equal number of features, and depending on the variance explained we extract it out. Similarly for LDA, the number of classes minus one, factor gets extracted. It essentially reduces the dimension, calling it a dimensional reduction process.

So does this mean, dimensional reduction techniques like PCA and LDA lack the explanatory components on determining which features are influencing the factors? If so, do there exist any other techniques which work in a similar way to exploratory factor analysis, rather than just dimensionally reducing the data?

Answer 1

You can use the principal components (PCs) in PCA in the same way as the factors in factor analysis (FA). If you express the PCs as linear combinations of the features, you can see how the features are "influencing" the factors. And FA is dimension reduction, too. Just consider the factors of FA spanning your linear (dimension-reduced) subspace in the same way as the PCs from PCA.

With LDA it is similar. You create a linear subspace that is spanned by vectors which you can interpret similar to the factors in FA. If you express those vectors with the original features, you see what feature combination is "influencing" the classification of your data the most.

In fact, PCA can be understood as a special case of FA. Both FA and PCA presume the same model: $$ \mathbf{y} = L\mathbf{x} + \boldsymbol{\epsilon} $$ where $L\in\mathbb R^{n\times d}$, $\mathbf{x}\sim N(0, \mathbf 1), \mathbf x\in\mathbb R^d$, with $d$ the dimension of the dimension-reduced linear subspace, $\mathbf y\in\mathbb R^n$, with $n$ is the number of your features, and $\boldsymbol\epsilon\in\mathbb R^n$ is the noise. The only difference between FA and PCA is the distribution of the noise: $$ \begin{align} \mbox{PCA}&: \boldsymbol{\epsilon_{PCA}}\sim N(\mathbf 0, \sigma^2\mathbf 1)\\ \mbox{FA}&: \boldsymbol{\epsilon_{FA}}\sim N(\mathbf 0, \mathbf D). \end{align} $$ where $\mathbf D$ is a diagonal matrix $\mathbf D = diag(\sigma_1^2,\ldots,\sigma_n^2)$.