This is the basic setup for a two-part model (TPM). It is useful when your outcome looks like the Loch Ness monster:
Lots of customers do not buy anything, so the substantial mass point at zero makes a single index model problematic. TPM relaxes the assumption that excess zeros and positives come from the same DGP.
Given a non-negative outcome $y,$ and an exogenous treatment assignment variable $t \in \{T,C\}$, we have
\begin{equation}
\overbrace{\mathbb{E}(y \mid x,t)}^\text{# of Purchases or Revenue} = \underbrace{\Pr(y>0 \mid x,t)}_\text{Buy or Not? (EM)} \cdot \overbrace{\mathbb{E}(y \mid y>0, x,t)}^\text{Then How Much? (IM)}
\end{equation}
Treatment can alter both the extensive and intensive margins. It can move customers from the head to the hump (EM), and it can move the hump further to the right (EM). The EM is modeled using a probit or logit model on the full sample. An LPM is another alternative. The IM uses robust Poisson for number of purchases, a GLM wih log link and inverse Gaussian or Gamma family for revenue, or even vanilla OLS with logged revenue, for those who actually bought something. In principle, you could have different covariates $x$ for each part.
You can look at the marginal effects at each margin, but you can also combine the estimates into overall average marginal effects:
\begin{align}
AME_{Levels} &=\frac{1}{N} \sum_{i=1}^N \left( \widehat{\mathbb{E}(y_{i} \mid t_{i} = T)}-\widehat{\mathbb{E}(y_{i} \mid t_{i} = C)} \right) \\
AME_{\%} &= \frac{1}{N} \sum_{i=1}^N \left( \frac{\widehat{\mathbb{E}(y_{i} \mid t_{i} = T)}-\widehat{\mathbb{E}(y_{i} \mid t_{i} = C)}}{\widehat{\mathbb{E}(y_{i} \mid t_{i} = C)}} \right)
\end{align}
These AMEs combine both intensive and extensive margin effects. I dropped the covariates from the equations above to reduce notational clutter, but they are implied.
In Stata, there's user-written command tpm by Federico Belotti, Partha Deb, Willard Manning and Edward Norton that will handle the hypothesis test on customer-level cross sectional data, including covariates :
tpm revenue i.treat x y z, first(probit) second(glm, family(igaussian) link(log)) robust
margins, dydx(treat)
margins, eydx(treat)
The output of margins might look something like this:
------------------------------------------------------------------------------
| Delta-method
| ey/dx Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
treat |
Treatment | .1411529 .0583387 2.42 0.016 .0268112 .2554945
------------------------------------------------------------------------------
That's corresponds to a 14% lift with a standard error of 6%.
Here's an attempt to hack something like the tpm marginal effects with some medical expenditure data. We will look at the two MEs of having health insurance, one on in dollars and one an elasticity (in %). The expected effect is negative. The first step defines a program that estimates the probit and GLM separately, and then calculates the marginal effect by hand by setting the insurance variable to 0 for everyone and then to 1 for everyone, which gives the marginal effects for each observation. I think you should be able to do something like this in R or Python:
set more off
use "http://cameron.econ.ucdavis.edu/musbook/mus16data.dta", clear
capture program drop mybs
program define mybs, rclass
set more off
preserve
gen insurance = ins
probit ambexp age i.female educ i.blhisp totchr i.ins, nolog
replace ins = 0
predict double phat0
replace ins = 1
predict double phat1
replace ins = insurance
glm ambexp age i.female educ i.blhisp totchr i.ins if ambexp>0, family(gamma) link(log) nolog
replace ins = 0
predict double yhat0
replace ins = 1
predict double yhat1
gen double delta = phat1*yhat1 - phat0*yhat0
sum delta, meanonly
return scalar delta = r(mean)
gen double lift = (phat1*yhat1 - phat0*yhat0)/(phat0*yhat0)
sum lift, meanonly
return scalar lift = r(mean)
restore
end
set seed 123456
bootstrap delta = r(delta) lift = r(lift), reps(1000) nodots saving(bs, replace): mybs
tpm ambexp age i.female educ i.blhisp totchr i.ins, first(probit, nolog) second(glm, family(gamma) link(log) nolog)
margins, dydx(ins)
margins, eydx(ins)
The bootstrapped results are:
. bootstrap delta = r(delta) lift = r(lift), reps(100) nodots saving(bs, replace): mybs
Warning: Because mybs is not an estimation command or does not set e(sample), bootstrap has no way to determine which observations are used in calculating the statistics and so assumes that all observations are used. This means that no observations
will be excluded from the resampling because of missing values or other reasons.
If the assumption is not true, press Break, save the data, and drop the observations that are to be excluded. Be sure that the dataset in memory contains only the relevant data.
Bootstrap results Number of obs = 3328
Replications = 100
command: mybs
delta: r(delta)
lift: r(lift)
------------------------------------------------------------------------------
| Observed Bootstrap Normal-based
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
delta | -179.972 93.18576 -1.93 0.053 -362.6127 2.668751
lift | -.103458 .0611306 -1.69 0.091 -.2232717 .0163557
------------------------------------------------------------------------------
These are the marginal effects from tpm on the original data:
. margins, dydx(ins)
Warning: cannot perform check for estimable functions.
Average marginal effects Number of obs = 3328
Expression : tpm combined expected values, predict()
dy/dx w.r.t. : 1.ins
------------------------------------------------------------------------------
| Delta-method
| dy/dx Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1.ins | -179.972 89.62025 -2.01 0.045 -355.6244 -4.319519
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.
. margins, eydx(ins)
Warning: cannot perform check for estimable functions.
Average marginal effects Number of obs = 3328
Expression : tpm combined expected values, predict()
ey/dx w.r.t. : 1.ins
------------------------------------------------------------------------------
| Delta-method
| ey/dx Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
1.ins | -.1099748 .0673351 -1.63 0.102 -.2419491 .0219995
------------------------------------------------------------------------------
Note: ey/dx for factor levels is the discrete change from the base level.
