1

I run an AB test and I have results at city level (keep in mind I run a one-sided test):

city,diff_means,p_value,ci_lower,ci_upper
 TGN,       0.1,  0.001,    0.09,    0.11
 BCN,       0.0,   0.78,    -1,1
 REU,      -0.1,   0.99,   -0.11,   -0.09
...

I run the Benjamini-Hochberg procedure and find a new significance cut-off (using 5% FDR), which is 0.01. Therefore, I select all cities that have a p-value of 0.01 or less and decide to apply the treatment on them.

In that case, what is the best estimate of the impact of the rollout (both point estimate and confidence interval)? I believe if we just take cities that we decide to rollout and run the analysis in them will give an optimistic (biased) estimate of the impact.

DrJerryTAO
  • 1,514
David Masip
  • 247
  • You are likely to overestimate the effect size. It's called the M-type error. See for several questions about it https://stats.stackexchange.com/search?q=type+M+error The M-type error is a bit difficult to compute. It depends on the true effect size, which is unknown. It is more an hypothetical value. – Sextus Empiricus Feb 25 '24 at 23:38
  • The over-estimation of effect size is a consequence of the selection on the basis of p-value, which is equivalent to selection on the basis of (scaled) effect size. Nonetheless, the observed effect size is still your best estimate of the effect size... Statistical test error rates and evidence do not play well together as they are in different universes. See here for more: https://link.springer.com/chapter/10.1007/164_2019_286 – Michael Lew Feb 27 '24 at 21:05
  • @MichaelLew better according to what standard? Wouldn't a shrinkage estimator have a lower squared error. – Sextus Empiricus Feb 27 '24 at 21:19
  • I guess so, maybe. – Michael Lew Feb 28 '24 at 04:31

1 Answers1

3

You are correct that selecting cities below a $p$-value threshold, like $p = .01$ in your example, and re-running the analysis on just those cities would give you an optimistic estimate of the treatment effect. The most common names for this effect is "selection bias" though other terms like "the winner's curse" are also used.

To see this quickly, imagine taking a thousand samples from a normal distribution $X_i \sim N(0,1)$, taking the 10 most positive observations, and taking the average of only those ten observations. This average will clearly be much larger than 0, the true mean.

Unbiased Estimation

Assuming you fit a model to the whole sample to select the cities as you described, the answer is that you can't get an unbiased estimate without making assumptions and the methods will be assumption-specific.

Taking a step back, you can get an unbiased estimate if you change your approach. The simplest way is to split your data into two data sets, and then use one data set to select the cities that get treatment and estimate the treatment effect in the selected cities using the other data set. The point estimates will be unbiased and you can construct confidence intervals with the correct coverages without any special methods. While simple, this isn't the most efficient use of all the data in the sample. There are numerous methods that build on this general approach but make more efficient use of the sample such as by repeating this procedure and combine the results in a way that preserves the unbiasedness property.

Related Literature

Three veins of work you can look for on this problem are "sample splitting", "post-selection inference", and "double/debiased machine learning" though there there are others. The literature on sample splitting is likely the least technical place to start because the other two settings often concern complications like high-dimensional problems. The problem is related to type M error as Sextus commented, but that concept is more for analyzing study designs since it involves considering hypothetical effect sizes (and sample sizes if done while planning a study).

Here are a few review paper:

Kuchibhotla AK, Kolassa JE, Kuffner TA. Post-selection inference. Annual Review of Statistics and Its Application. 2022 Mar 7;9:505-27.

Kennedy EH. Semiparametric doubly robust targeted double machine learning: a review. arXiv preprint arXiv:2203.06469. 2022 Mar 12. (pdf link)

Fithian W, Sun D, Taylor J. Optimal inference after model selection. arXiv preprint arXiv:1410.2597. 2014 Oct 9.(pdf link) And an early paper on the subject that's incredibly short and readable: Cox DR. A note on data-splitting for the evaluation of significance levels. Biometrika. 1975 Aug 1;62(2):441-4.

John Sperger
  • 96
  • 3