Model causality: graphical models and PCA

Question

If we build a graphical model (DAG) we (may) interpret the arrows as causal dependences.

If we build a graphical model based on the variables returned by principal component analysis (PCA) we should obtain a totally disconnected graph (assuming normality). We could interpret this as the variables given by the principal components having no causal dependency and being attributed to different (latent) causes.

How are these interpretations of the data compatible ? Is the concept of causality dependent on the set of variables chosen as a bases?

ps: for easiness of interpetation we may suppose that all variables have the same unit of measure.

Answer 1

PCA makes no statements about causality; it merely identifies (linearly) uncorrelated factors of variation. That two variables are (linearly) uncorrelated does not imply that they cannot be causally related, so if you want your graphical model to represent causality, then you basically cannot conclude anything about the presence or absence of edges from a PCA.

--

Let us consider a simple example. You have two variables, $A$ and $B$. $A$ follows a standard normal distribution, and $B$ is caused by $A$ following $B:=A+\varepsilon$, where $\varepsilon$ follows another standard-normal distribution. Thus, B is also normally distributed with mean 0 and variance 2, and $A$ and $B$ together are jointly normally distributed, as (I understand) you assumed in your question. If one intervenes on $A$, that will change $B$, but if one intervenes on $B$, that will not change $A$.

Firstly, notice that purely from observations of the joint distribution of $A$ and $B$, it is impossible to conclude anything about their causal relationship! $A$ could be causing $B$ (as we assume it is), $B$ could be causing $A$, or both could be caused by a common confounder. (Also see the nice example on Jonas Peters' homepage in this regard.)

Now you perform a PCA, which will give you the principal vectors $PV_1=(1, 1)/\sqrt{2}$ and $PV_2=(1,-1)/\sqrt{2}$, associated with the principal components $PC_1=A+B$ and $PC_2=A-B$. The correct causal diagram would now be that both $PC_1$ and $PC_2$ are caused by both $A$ and $B$, which follow whatever causal relationship was true for them in the first place. In essence, it seems to me that the principal components will always be descendants of shared confounders, namely all the original variables.

--

Inferring any kind of causal relationships from purely observational data is an extremely hard task and generally impossible without imposing additional assumptions ("I know for a fact that $B$ cannot cause $A$, and I also know for a fact that there cannot be any third thing influencing both of them."). For a great overview on the assumptions one can make to achieve this, I recommend Jonas Peters' book (PDF available for free under the link). It is, however, very much a topic of current research and essentially unsolved. In this regard, I also found the historical struggle to prove something as "obvious" as that smoking causes cancer quite instructive. ([1], [2], [3], [4])

Answer 2

1. The principal components you obtain are orthogonal (uncorrelated) among themselves by definition (of PCA), which is already not a great start if you want to assume some kind of dependence among them.
2. Besides that, they do not carry a natural meaning from the perspective of your domain of study (only statistical) except the one you give them.

Maybe you can include principal components in a DAG, but in practice you will have to make several assumptions and in general I really do not think is a good idea (I am very much open to counterexamples). On the other hand, you could infer relationships (correlation, non causation) among the original variables computing the correlation coefficient between them and the principal components. From the interpretation of these correlations potentially you could add new nodes (your principal components) in your original DAG as latent variable with arrows into the variables they mostly correlate with.

To answer this part of the question:

Is the concept of causality dependent on the set of variables chosen as a bases?

The concept of causality depends mainly on you and your situation. Causal inference can be very useful, but I think its reasoning is not unique (except in some straightforward cases). As follows a partial and succint explanation of why (in my opinion):

First: Can I find a DAG for all situations? The concept of causality is always highly dependable on the researcher knowledge and the data that are available. On top of these two things, building a DAG is dependent on your research question. Unless you are looking into a deterministic problem (for example in physics), you cannot build a "complete" DAG the stands in any possible situation (any research question) involving these variables.

The book What If by Hernan and Robins is a simple introduction* to the DAGs world using a perspective from structural causal inference, where you can find some not too complicated mathematical proofs of their reasoning. Second: using my own DAG. The theoretical framework described in the book provides you with some ready-made statistical tools to investigate your data based on the DAG you define, but eventually even in this case I almost always end up using different statistical tools (such as PCA) to back up my reasoning and get new insights.

Just so you know, this relationship between correlations and orthogonality is not widely accepted here

*There are other views and styles you might like better, but in this book they really take it slow and eventually you get quick comprehensive picture on several topics. On the other hand, if you are more machine learning oriented I recommend this.