Is it valid to look at the impact of a feature on residuals?

Question

Background

I'm trying to measure the causal impact of action on outcome (sorry for the vague names, but trying to keep this general). My data consists of the following for each record:

• m other_features [the result of TruncatedSVD to a much larger number of features]
• action which is a binary
• outcome which is continuous [in all cases here, I use the log of the actual value]

Based on business knowledge of the situation, I'm confident in a causal model that states:

• the other_features have a causal effect on action
• the other_features also have a causal effect on outcome
• the action have a causal effect on outcome

I tried two techniques and got different results - I wanted to know how to interpret these results.

Attempt 1: Vanilla Linear Regression

First, I tried to regress outcome against other_features and action. When I did this, I got the following results (I'm using $\beta$ to refer to linear coefficients):

• $R^2 = 0.793$
• $\beta_{action} = 0.0943$

This implies that the action has the effect of ~9.5% increase on outcome.

Attempt 2: Looking at Action vs Residuals

Second, I tried to regress outcome against other_features (i.e. without action). When I did this, I got the following results:

• $R^2 = 0.793$ [very little difference with the original regression).

Then, I looked at the residuals of those with and without action and got the following:

This seems to indicate that action has about a ~-8.3% impact on outcome.

Questions

1. Is attempt 2 a valid way to proceed?
2. If yes, how should I interpret the difference between the two approaches?
3. Is there anything I ought to be weary/careful/aware of in using this approach?

(Please let me know if it additional data/results would be helpful in interpreting these results, I can supplement the information here).

Answer 1

Your question has nothing to do with causality per se, it is actually about the mechanics of OLS. So let me leave the causal part to the end.

Thus, first, regarding the two approaches of fitting an OLS model, your second approach of regressing residuals on the variable of interest is related to what we call the FWL theorem.

Your only mistake is that you also need to regress "action" on "other_features", and only then you would regress the residuals of "outcome" on the residuals of "action". As it is, your "attempt 2" is not correct, since it is not fully adjusting for your "other_features". If you residualize both treatment and outcome, then "attempt 2" will give you numerically the same results as your "attempt 1" (barring corrections on the actual degrees of freedom).

Now, regarding the causal interpretation, first you need to decide whether "other_covariates" satisfy the back-door criterion. Second, you need to make sure a linear model is good working approximation, otherwise linear regression will not do the proper adjustment (or you need to reinterpret the coefficient as weighted average of causal effects, see Mostly Harmless Econometrics, Chapter 3).

Answer 2

Trying to establish causality is the hardest part of statistics. The short answer is no, you cannot be confident you've established causality, unless $action$ was randomly assigned. If you have customers coming to your business, and some of them have a 0 and some have a 1, that doesn't necessarily imply imposing 0 or 1 on random individuals will have the effect reported from a statistical model. Same holds if it's your business assigning the action to customers, unless you just blindly take the action, with no other considerations (A/B testing). Most likely the action is being done due to some expectation the action will be productive, i.e. $action$ is correlated with some features, seen or unseen, of the customer. If you do have randomized assignment, then your linear regression approach should pick up the causal effect in the $\beta$ coefficient. If you just want to make predictions, given that we know someones $action$ status, we can make some estimate about his outcome.