Paired t test-like solution in a two-group pretest/posttest design

Question

I am analyzing a dataset where pre/post measurements are made (before/after a surgery) on patients who can belong to two groups, and we are interested in how group membership - let's call it G - affects the impact of the surgery. (It is a nonrandomized observational study, but let's put other confounders aside now for simplicity.)

I am no expert in such situations, so I read a bit, and it seems that percentage change is a bad idea. As this is not a randomized trial, it is also not a question that we can not neglect the baseline measurement. Thus, we are left with two options:

lm( I(Post-Pre) ~ G, data = RawData )

(Intercept)            G  
   -0.02563      1.04204 

i.e. change score, or

lm( Post ~ Pre + G, data = RawData )

(Intercept)          Pre            G  
  -0.003125     0.006014     0.973866  

i.e. baseline adjusted as a covariate. (That is, ANCOVA, if I am not mistaken with the terminology.)

What do they mean, and how they are related to each other is discussed in the literature in great detail. My understanding is that generally speaking the latter is the better approach. Hope I am right up to this point.

Now, what I don't understand is how paired $t$ test fits into this picture. That is, how

lme( Meas ~ PrePost * G, random = ~1|id,
          data = reshape( RawData, varying = c( "Pre", "Post" ), v.names = "Meas",
          timevar = "PrePost", direction = "long" ) )

(Intercept)     PrePost           G   PrePost:G 
 0.02264409 -0.02563254 -0.06858446  1.04203755 

Random effects:
 Formula: ~1 | id
        (Intercept) Residual
StdDev:   0.0802519 1.034537

relates to the above procedures...? (I called it "paired $t$ test-ish" because lme( Meas ~ PrePost, random = ~1|id ) would have been equivalent to the usual paired $t$-test, with no group membership.)

I don't think that it is equivalent to any of the above approaches, and it might be completely pointless (I have the feeling that it measures something different), but I simply don't see what it means...

(The above examples pertain to set.seed( 1 ) and RawData <- data.frame( Pre = rnorm( 1000 ), Post = rep( 0:1, each = 500 ) + rnorm( 1000 ), G = rep( 0:1, each = 500 ), id = 1:1000 ).)

Answer 1

The paired t-test is best applied to a crossover study so that order effects can cancel out. For observational pre-post comparisons the design is very weak.

The ANCOVA on the raw response is preferred because the slope on pre might not be 1.0 and the difference post-pre may not be ordinal if post and pre are ordinal but not interval scaled. Ordinal regression is a general solution, and must be done on raw values.

Your ANCOVA example is comparing 2 independent groups but the paired t is testing a completely different within-group hypothesis. I'm not as clear on the random effects model. You'll see old discussions of this in the statistics literature under the phrase "recovery of intra-block information".

Answer 2

I think the complaints brought up by the Vanderbilt site are valid ones, but they can't be applied universally without knowledge of the question. They mention specific graphical tests which can be used to assess assumptions about a magnitude of effect. For instance, in HIV rebound patients, administration of antiretroviral therapy (ART) has a demonstrable geometric mean effect. That's because when the virus is more prolific, the PK dynamics are such that the virus interacts at a higher rate with the ART. Issues with interpretation behoove us to think more carefully about our interpretation than to seek in vain an omnibus that permits us to be loose with interpretation.

You bring up two separate points: ANCOVA vs paired differences and percent change as the scale of effect. Modeling change as a percentage involves performing a log transform of the outcome, re-exponentiating the coefficients obtained from the linear model, and interpreting them as a percentage change. A paired t-test is a special case of a linear model without log-transform where you actually model an offset:

\begin{equation} E[Y_1 | Y_0, X] = \beta_0 + Y_0 + \beta_1 X \end{equation}

$\beta_1$ is the intervention effect as a mean difference obtained from a paired t-test, or a univariate t-test based on pairwise differences in $Y_1, Y_0$. Note the term $Y_0$ does not have a coefficient.

The ANCOVA is seen as a better model because it is a more flexible parametric model which incorporates a coefficient for the referent/baseline term.

\begin{equation} E[Y_1 | Y_0, X] = \beta_0 + \gamma Y_0 + \beta_1 X \end{equation}

$\beta_1$ still has the same interpretation: the expected difference in outcome comparing individuals who receive the intervention to those who don't conditional upon baseline/referent value.

To model percent change in an ANCOVA design, you apply log transform of the outcome.

\begin{equation} E[\log Y_1 | Y_0, X] = \beta_0 + \gamma Y_0 + \beta_1 X \end{equation}

And $\exp(\beta_1)$ is interpreted as a percent difference in outcome comparing those who receive intervention to those who don't conditional on their baseline/referent value.

The issue of homogeneity which is addressed in the Vanderbilt site is also worth mention. Suppose you are interested in modeling percent change as with our final log transformed ANCOVA model. Suppose further, there is a truncation of treatment effect for a range of values, like with our HIV patients receiving ARTs. If the viral load drops below a threshold the devices cannot detect its presence. In many data analyses, analysts impute the minimum (a pragmatic and inefficient solution). This would, by assumption, violate the predicted trend. However, we can view the model as summarizing those data in spite of those differences, or any other subgroup differences, when we interpret the $\beta_1$ or $\exp \beta_1$ as the Average Causal Effect (or ACE). This is an issue of interpretation that considers the whole population rather than what individual outcomes are, which is often a follow-up question requiring much more extensive design and investigation.

The mixed model is a bit confusing since you haven't defined terms. Nonetheless, a mixed approach is the way to go if there are more than one post measurement. Why would you assess more than one post measurement? Measurement error: blood pressure? different every day, viral load? different every day, weight? you get the picture. By design if an outcome can be measured cheaply but has low reliability, an efficient design might specify multiple measurements as a way of reducing intraindividual variability. However, in doing this, we now have correlated measures. Introducing a random intercept further reduces intraindividual variability, and hence the overall variability of the effect estimate, by inducing an exchangeable correlation structure where each repeated measure within an individual has a fixed, constant, positive correlation. This design feature updates ANCOVA to be a "repeated measures ANCOVA".

As a note your R notation is wrong. Note in my describing the above design: we incorporate a baseline measure as a covariate in the ANCOVA design or an offset in the paired design. In your reshape dataset, the pre and post measures are taken as exogenous measures which are predicted by G which I presume is the intervention design. Baseline measures are NOT predicted by this value, so that model is wrong. "Repeated measures" in a pre-post design means you have more than one measure of the outcome.