1

Background

Let’s rewind time and say a company like Twitter wanted to introduce a feature that allows each user to add a banner to the top of their profile page. After creating this feature, 5k beta tester high-profile accounts were auto-enabled into the feature and then 5k other accounts organically enabled it.

Experiment

We want to design an experiment that sees if this banner impacts some engagement metric such as “Total Favorites” from each user that sees it.

To structure this experiment, we do a visitor-side randomization where visitors are exposed to the experiment when they visit any of the ~10k profiles that have the banner.

Test: Participants can see the banner if they visit a profile with a banner

Control: Participants can’t see the banner if they visit a profile with a banner

Outcome Variable: Total Favorites for each participant

What I'd like to understand

  1. How the total number of visits to an auto enabled profile with a banner impacts Total Favorites I.e. What’s the relationship between how many times they saw a banner and the impact on total favorites

  2. How the total number of visits to an organically adopted profile with a banner impacts Total Favorites. I would want to use this to understand the impact on favorites from buyers interacting with the organically adopted profiles. I realize there is a bias with who has organically adopted it, but would like to understand even with the bias.

My initial solution would be a regression model similar to below:

Total Favorites = α + β1 (enabled_high_profile_visits) + β2(organic_enabled_profile_visits) + β3 (test_treatment) + β4 (test_x_enabled_high_profile_visits) + β5 (test_x_enabled_high_profile_visits)

Definitions

enabled_high_profile_visits - total visits to profiles that had the unit auto-enabled organic_enabled_profile_visits - total visits to profiles that organically enabled the unit test_treatment- binary condition if you are part of the test group, and therefore have been exposed to a profile with the banner

My concern is that I will run into post-treatment bias by conditioning the regression on post-treatment variables. Is there any alternative solution, or is this not a problem in this case?

Alex
  • 55

1 Answers1

2

I may not fully understand some of the concepts like organically adopted and auto enabled but I will try to answer the question nonetheless. I will talk in terms related to clinical trials since that is a similar randomized setting I am familiar with. Conditioning on post-treatment covariates can be seen as a subgroup analysis among a set of "responders" or "non-responders". If you are interested in the treatment effect for the entire target population you should not condition on these post-treatment covariates. If you are interested in these subpopulations, then definitely condition.

In a clinical trial setting we are often interested in inference for the entire target population. In the trial subjects will often switch or discontinue treatment as a result of being a "responder" or "non-responder." One option is to imagine a world where switching would not occur when defining the estimand (e.g. population-level mean). This would amount to censoring subjects after a treatment switch and using an unverifiable missing data assumption to account for their missing values during the analysis. Another option is to incorporate this switching into the treatment definition, i.e. a treatment policy d(A,B):=[start with treatment A; if it doesn't work switch to treatment B]. This amounts to averaging together resonders and non-responders (treatment switchers and non-switchers). For this estimand causal inference remains at the policy level.

It sounds like you might also be worried about sampling bias $-$ that the sample of subjects in your experiment are not randomly selected in a way that is representative of the target population. You can use inverse-probability-weighting or propensity scores to re-weight the sample in a way that you think is representative of the population. This is typically an unverifiable assumption and should be accompanied by a sensitivity analysis. An easier approach is to acknowledge potential bias and use a much smaller significance level (larger confidence level) when performing inference. This doesn't remove the sampling bias, but would help to ensure the confidence interval covers the truth despite the bias.

Apologies again for not using your same terminology. If you refine your question based on my answer I can make an attempt at improving my answering.

Geoffrey Johnson
  • 2,800
  • Thanks Geoffrey.

    Let's say we are testing a drug which ends up lowering blood pressure. However, one of the consequences of the treatment is that it also increases the likelihood users also take vitamins (sorry for the lack of imagination) and this also lowers blood pressure.

    If you would like to isolate the impact of the treatment vs the impact of the vitamin usage, would it make sense to condition on the vitamin usage?

    What are the precautions I need to take here? A lot of the literature I've read warns against controlling for post-treatment variables in regressions:

     – Alex Oct 31 '21 at 22:07
  • Example link: https://p-hunermund.com/2018/08/27/why-you-shouldnt-control-for-post-treatment-variables-in-your-regression/ – Alex Oct 31 '21 at 22:08
  • Great question! If you condition on a particular treatment as well as post-baseline vitamin useage, the resulting estimation and inference is a subgroup analysis akin to conditioning on covariates like sex or race. If you want to isolate the impact of treatment it would be more appropriate to censor the outcome for subjects who take vitamins. If you make a missing at random given treatment assumption for these censored observations, the resulting analysis should look like an analysis conditioned on no vitamins. However, by censoring we could entertain other missing data assumptions. – Geoffrey Johnson Nov 02 '21 at 23:07
  • By censoring we imagine a world where vitamin usage would not occur. I recommend reading papers on the ICH E9 addendum on estimands. This is for clinical drug development. This document is great because it encapsulates causal inference using the jargon of a clinical trialist. I have found that the jargon used by most in the causal inference literature can be a little difficult to grasp. – Geoffrey Johnson Nov 02 '21 at 23:10
  • If the treatment assignment is randomized, then each treatment group should be representative of the entire sample, which in turn should be representative of the entire target population under investigation. When we condition on treatment it is still representative of the entire population. If we further condition on vitamin usage there is no guarantee the sample in this subgroup is representative of the entire target population. We can assume so, but this assumption should be made clear. – Geoffrey Johnson Nov 02 '21 at 23:28