In a diff-in-diff or regression discontinuity research design, why is it important to describe why the counterfactual is a plausible one?

Question

I've heard it mentioned that in difference-in-differences, regression discontinuity, or even in some other quasi-experimental research designs, that the counterfactual should be explained as a plausible outcome. I'm feeling dumb and can't wrap my head around why that's important though. Is it to explain that without the research, the counterfactual would have happened and its implications? Isn't it basic intuition that treatment relates just to the treatment group? Any explanation would be much appreciated.

Answer 1

It’s a natural human tendency to interpret events in causal terms. As humans, we also tend to talk about paths not taken, and what would have happened had we chose that path in life. For example, I’m the youngest in my family. I didn’t always receive the undivided attention of my parents; they grew weary raising my older brothers, so you could say they were a bit lazier with me. To overcompensate, I worked a little harder; I did well in school. I did this to garner the attention of my parents; I wanted them to be proud of me. Over coffee one morning, a colleague of mine exclaimed, "...the youngest child is always the most successful." In other words, I outperformed my older siblings because I was the lowest in birth order. Had I been a middle child, I’d be in prison.

But that’s balderdash, to some degree. To test this causal claim, we should go back in time and measure my achievement from birth, to the present day, as a middle child. Sounds stupid, right? It’s already too late! In the present state of the world, I am not a middle child. The “fact” of the universe is I’m “the baby” of the family. To claim that being lower, or lowest, in birth order makes you more conscientious or successful, we should compare my life outcomes to a person as similar to me as possible but for our birth order. The individuals “like me” but higher up on the birth order food chain represent the counterfactual. They represent a state of the world that is counter to the factual realm. In essence, you should try to find a near perfect clone of me, except he's a middle child. That clone is a "parallel version" of me that went out into the world as a middle child. Assuming we actually find a person like me, we could compare my achievement records with that of my near perfect clone.

The foregoing example is a bit farcical, but people make claims like this all the time. Have you ever heard people utter some version of the following:

Had I attended a private university, I'd be far better off financially than I am now.

Had I been an only child, I'd be way more self-absorbed.

Had I been born without a father, I'd probably be in a gang.

Had I never gotten married, I’d be much more happy right now.

Note, these are statements about cause and effect. Technically, these examples simplify the world, to some nontrivial degree, by looking at singular causation. In other words, "if" we could simply change/manipulate $X$ (treatment), then $Y$ (outcome) would be different. Again, we are talking about states of the world that did not happen. And, even though they didn't happen, we're prone to making statements about how things would have been different had we only changed that one thing.

Let’s look at one other example. As a policymaker, say I'm interested in reducing the number of homicides in a select few counties throughout the United States. I "crackdown" on the violence by introducing a law enforcement intervention. During the evaluation phase, I track reported homicides before and after the crackdown. Homicides abate; I declare success. Simple, right? Not really. I more than likely introduced a policy in areas with a demonstrable need. Should I worry about regression to the mean? Probably. Maybe a cessation in violence was going to happen anyway. How can I make valid claims that my treatment actually caused homicides to go down? The best way is to hop into a DeLorean, go back in time and simply take away the intervention. Then, I measure crime outcomes under both states of the world. But not so fast. We only really observe one "factual" state of the world.

To buttress causal claims, I should find jurisdictions that "look like" the treated counties, but had never been exposed to the intervention. Maybe some counties were "almost considered" for treatment but didn't make the cut. The cohort of untreated counties should look as similar as possible in terms of covariate balance to the treated counties. Homicides should also "trend" similarly between treated/untreated areas before the intervention. If the inter-temporal evolution of the homicide trends held in the past, then it's reasonable to assume they will hold in the future. If we can demonstrate this, then the untreated areas represent that state of the world in the absence of the intervention. In other words, the counties unexposed to the intervention represent an "approximation" of what would have happened had I never introduced a treatment. That's the counterfactual.

Answer 2

In a quasi-experimental design, the comparison group is constructed to look as close as possible to the treatment group just prior to the intervention. All covariates are balanced in about the same proportion as the treatment design. And if you randomly selected 1 person from the treatment and comparison group, it would be difficult to determine who was a member of the treated group and who was not, based upon the before treatment variables.

The only discernable difference is that the comparison group did not go thru the treatment program. Some members of the comparision group may also have had the same outcome as the treatment group, and some may not. But the critical point is that is you can show statistically that enough of the comparison group had the same positive outcome as the treatment group, that would be consider the counterfactual, and there would not be enough evidence for treatment success.