Practical usefulness of PCA

Question

I asked a similar question in the past, but I've thought about the message I am trying to convey a bit more and feel I can articulate it better. For context, I am on an introductory course in machine learning. A while ago we covered PCA, and from my point of view, I really can't see what it's good for in the real-world, and Google searching doesn't seem to shed much light on my question.

To illustrate my confusion, imagine we have a big data set and we would like to run PCA on it. Suppose (for convenience) that using 2 principal components explains an adequate amount of the variability. Now what? All we have information on now is the linear-combinations of a subset of the variables. I feel like at this point the data has become too abstracted to be interpretable in any meaningful way, in general. So, I reiterate, what's the point?

I might be missing something as PCA is used in industry all the time. My ignorance probably comes from only being on an introductory course. Does anyone have any perspective that might be useful?

Answer 1

One important use of PCA is in analysis of electroencephalography (EEG) data. To measure an EEG, dozens of electrodes are attached to your scalp and measure electric currents in your brain, either at rest or while you perform some experimental task.

Of course, the measurements at neighboring electrodes are heavily correlated, because they are generated by activity at a specific region in the brain, which then creates electric currents that will be picked up by all electrodes in the vicinity. It's not easy to learn about what happens deep in your brain if all you have is measurements from your scalp, but for some reason, few people are fine with having deep electrodes driven into their brain.

One thus reduces the dimensionality of the problem using PCA, which in this particular application also has a temporal component. You are completely right that it is hard to actually interpret the principal components. However, over the decades a body of research has developed that lets us expect particular principal components loading on particular electrodes, with peaks at particular points in time, e.g., after being presented with a specific stimulus.

For instance, a long time ago I looked at the P300, an event-related potential that loads over the parietal lobe (there's your PCA) about 300 ms after presentation of a stimulus that requires some kind of decision. In this particular analysis, the experiment was about whether spider phobics and non-phobics reacted differently to drawings that could be interpreted as spiders. The (unconscious) decision whether a particular drawing "was" a spider elicited a P300, and that indeed differed between phobics and nonphobics. Using a PCA and analyzing the parietal principal component - instead of, e.g., a single specific electrode = allows reducing the noise in such a setting, by essentially averaging the signal from multiple electrodes.

Answer 2

First, from the perspective of education, PCA is a good entryway to the world of dimension-reduction techniques and associated methods. Whether we're talking ICA, non-negative matrix factorization, confirmatory factor analysis, partial-least squares, canonical correlation analysis... (you get the idea), understanding PCA gets you halfway there.

From a practical standpoint, there are lots of times when PCA can be sensibly used to combine variables - I think the other answers give good examples.

Now, I think what your question is really getting at is - what's the point of a naive application of PCA? What's the point of throwing PCA at a long list of variables and getting back a handful of components? Well, it's quite useful if all you care about is model prediction, and it's also quite handy when you have a lot of highly correlated variables in your data set. PCA is also very useful if you want to understand the structure of your data - for example running PCA on genetic data (millions of variables) tells you about broad trends in ancestry in your data set.

Answer 3

I use three examples in my lectures to illustrate what PCA can do (click the links for pointers to the slides). They're chosen to show how useful it is in general data science practice and how powerful it can be (especially given that it's just a linear transformation).

example 1: bone analysis

Imagine you're a paleontologist, and you find a shoulder bone. Because of years of training, you recognize immediately that this is a Hominid bone (which is very rare) and not a Chimp bone (which is common).

How do you report that fact? "It's a Hominid bone because I say so", isn't very scientific.

One way would be to use PCA. You measure a bunch of features on the bone and on some similar bones, and you plot the first two principal components. Here's one example of such a plot [1].

The plot very clearly shows what your trained eyes told you in the first place: the bones of Hominids cluster together, far away from the more common bones of humans, chimps and other apes.

You can even draw a line backwards through the known evolutionary path, to get a hypothesis where humans come from. Turns out chimps and bonobos are a better candidate than gorillas and gibbons.

example 2: DNA

Take about 1300 people in Europe, sequence their DNA, and check about half a million markers. This gives you a dataset with 1300 instances, and 500 thousand features. Apply PCA, and plot it by the first two principal components. Now color the points by where the subject is from. Here's the result [2].

The plot reveals that the first two principal components provide a blurry picture of the geographical distibution.

I admit it's hard to say what this is useful for, but it certainly illustrates the power of the method.

example 3: eigenfaces

Take a set of images of people's faces and flatten then into high-dimensional vectors. Run PCA.

Again, you will see clusters for certain meaningful concepts. But here, we can do something else cool. We can take the n-th principal component, and nudge one of the examples in our data a little in that direction.

Here is the result for the first couple of principal components:

1 Fossil hominin shoulders support an African ape-like last common ancestor of humans and chimpanzees. Nathan M. Young, Terence D. Capellini, Neil T. Roach and Zeresenay Alemseged http://www.pnas.org/content/112/38/11829

2 Novembre, J., Johnson, T., Bryc, K., Kutalik, Z., Boyko, A. R., Auton, A., ... & Bustamante, C. D. (2008). Genes mirror geography within Europe. Nature, 456(7218), 98-101. https://www.nature.com/articles/nature07331

Answer 4

One application of PCA that I have used a few times is the construction of social indicators. We use the projection of each observation (usually households) over a component axis (usually the first), and use it as an indicator for public policy.

This is possible because the surveys that we use are designed to capture that information. You can look online for "quality of life questionnaires".

Another thing to take into account is that PCA may not be the best method in many applications, but is used for "backward compatibility". If you change the method, you will be unable to compare with past measurements. And the result will not be useful for building public policy.

see: http://article.sapub.org/10.5923.j.statistics.20221203.03.html

Answer 5

It is absolutely not clear, obvious, or agreed-upon what "intelligence" is in humans (or, for that matter, in nonhuman animals). What is clear is that people's performance on a variety of tasks involving mental processes (a very general term) is highly correlated. If you speak five languages, then chances are that your performance on standard matrix tests is also above average.

The probably currently best accepted "theory of intelligence" essentially boils down to performing a PCA on the result of multiple tests of mental processes and calling the first principal component the g factor. After you have done this, you start arguing with other psychologists about what precisely you are measuring, whether there truly is an underlying trait "intelligence", and what later principal components are measuring.

This construction gets more interesting and relevant as artificial intelligence (or, cough, not) gets better and better.

Answer 6

I've used PCA in facial motion capture for real time animatronic control of the 'lots of dots on a face' variety.

I was able to find out which dots - which is to say regions of the face - encoded the most information in movement and emotive expression.

It's obvious to some where these may be, and there is a natural correlation between these areas and how much they move, but I wanted to confirm my intuition.

I could only track so many dots so with that information I was able to more efficiently place them around the face, with more density in areas that encoded the most useful 'perpendicular' data and more sparsely in those that only became relevant on their own merits occasionally.

This is grossly simplified, and there was a lot more to it (the eyes... another world going on there) and I'd probably use a NN or similar with no dots this time round, but PCA played an integral part of the learning.

Answer 7

PCA just comes down to using the eigendecomposition of the (empirical) covariance matrix of the data. The full eigendecomposition of the covariance matrix results in a set of eigenvectors and corresponding eigenvalues, which can be interpreted as the variance along these eigenvectors. Because these eigenvectors are orthogonal, they can be used to create a rotation matrix that rotates the data so that the basis aligns with the direction of maximum variance.

For some people, this may be easier to understand with a few lines of code:

import numpy as np
data = np.random.randn(256, 2)  # generate some random 2D data
covariance = np.cov(data, rowvar=False)  # compute covariance matrix
variances, rotation = np.linalg.eigh(covariance)  # eigendecomposition
pcs = (data - data.mean(0)) @ rotation  # compute principal components

In other words, principal components are just a rotation version of the (centred) data. This also means that the (centred) data can simply be reconstructed by rotating back the principal components: pcs @ rotation.T. As a result, a "full" PCA (not sure if this is the correct term) is perfectly reversible. This is how PCA is typically used in the context of pre-processing data. After all, this rotation should typically make it easier to find important features. Typically, the principal components are additionally whitened (scaled to unit variance) using the eigenvalues of the decomposition (variances in the code). It is also possible to whiten the data after the rotation and then rotate it back, which gives rise to ZCA pre-processing. I can highly recommend this answer to a related question for further reading (and some nice figures).

When using PCA for dimensionality reduction, you would only use a subset of the columns in the rotation matrix. This obviously leads to loss of information. However, the columns of the rotation matrix can effectively be to transform the data back to the original space. After all, using only a few columns corresponds to setting principal components to zero (i.e. dropping information). If the total variance corresponding to these dropped dimensions is low enough, a reasonable reconstruction (of the centred data) is typically possible.

Again, in code:

data = np.random.randn(256, 784)  # generate some random high-D data
covariance = np.cov(data, rowvar=False)  # compute covariance matrix
_, rotation = np.linalg.eigh(covariance)  # eigendecomposition
reduced_pcs = (data - data.mean(0)) @ rotation[:, -70]  # dimensionality reduction PCA
reconstruction = reduced_pcs @ rotation[:, -70:].T  # reconstruction of (centred) data

Note that the reconstruction of the random data in this snippet of code is not going to work well. However, you should get some reasonable results if you plug in e.g. some MNIST data.

TL;DR: PCA is often used to pre-process data (make it nicer to work with) and can actually often be transformed back to the original input space (i.e. is not necessarily abstract).

Answer 8

Lots of what we're trying to get at with dimensionality reduction, whether linear or nonlinear, is abstraction of the data away from a raw space and towards a manifold or embedding in which complex phenomena can be described with fewer, more meaningful parameters. PCA is arguably the simplest of these techniques; it will help you reduce correlated variables to a single common dimension. Implicit linear correlations in your dataset aside, lots of things in the real world are expressed as linear combinations of a set of latent variables. For example, you can use SVD (a generalization of PCA) in natural language because documents that contain words like "cat" are also likely to contain words like "pet" in rough proportion. So in a bag-of-words or tf-idf model, the correlation gets called out.

Aside from dimensionality reduction (or rather why it works for dimensionality reduction), PCA/SVD attempt to maximally explain the overall variance of the data using fewer components. You can think of this as "stretching" each component apart from the others as much as possible while keeping the component itself coherent (and this is actually explicitly how more advanced dimensionality reduction techniques such as maximum variance unfolding work). That makes it very useful at teasing apart how important each linear factor is within a dataset. That turns out to be particularly useful in recommendation engines over graphs, like the one that won the Netflix prize.

If PCA isn't useful today, it isn't because it inherently lacks usefulness, but because we have more powerful nonlinear techniques that have supplanted it.