5

I have a question on the assumption of exchangeability in permutation tests. Although I read a lot about this topic, I am still confused.

For $N$ subjects, I have the value of a clinical measure $Y$ (a numerical quantity such as the volume of a ventricle) together with a set of other clinical parameters (such as age, gender, height...) and a categorical variable which represents the presence of a genetic mutation. I would like to use permutation testing with the following multiple regression model:

$Y = \beta_0 + \beta_1 \; age + \beta_2 \; gender \; + \; ... \; + \;\beta_n \; mutation + \epsilon$

to see if $\beta_n \neq 0$.

The only assumption required by the technique I would like to employ (Freedman and Lane, 1983) is the one required by all the permutation tests: exchangeability.

If my understanding is correct I need to check if I can shuffle the values of $Y$ across the $N$ subjects under the null hypothesis (no effect of the genetic mutation) and this doesn't affect the error ($\epsilon$) distribution.

I believe that this is not true in this setting as Y depends also on the other parameters in the model (age, gender etc.), but I am not sure about that. I was wondering what you think about that and what I should check to correctly apply permutation testing in this setting.

lmo
  • 807
  • 1
  • 7
  • 17
Helm
  • 91
  • 1
    Please be explicit about what paper you mean. If you're talking about Freedman D. and Lane, D. (1983), "A Nonstochastic Interpretation of Reported Significance Levels," Journal of Business & Economic Statistics, Vol. 1, No. 4 (Oct.), pp. 292-298, ...

    then include a full reference, and outline what, in particular they're doing that you're trying to emulate (we shouldn't have to go read an entire paper to then hazard a guess as to what you are actually trying to do; you should explain what it is you're doing).

     – Glen_b May 23 '16 at 07:01

1 Answers1

5

Assuming you mean the paper I mentioned in comments, Freedman and Lane [1] explain it clearly.

The last sentence of the abstract for their paper* says:

This development parallels that of randomization tests, but there is a crucial technical difference: our approach involves permuting observed residuals; the classical randomization approach involves permuting unobservable, or perhaps nonexistent, stochastic disturbance terms.

*(which you can read without access to the journal, but I assume you do have access to the paper if you're trying to do what it does)

That says it all, pretty much.

What is exchangeable in regression is the error term, $\epsilon$. Note that under the null hypothesis the error term of interest is that for the model where Y is regressed on all the predictors but the one being tested (the reduced model).

You can't get at $\epsilon$. So they use the best available estimate of them, the residuals.

Those aren't really exchangeable (they don't have the same distribution and they're not equicorrelated). So that's then not quite a permutation(/randomization) test, and they clearly say so right there in that abstract -- they say it "parallels" the classical randomization test, which it does in a particular sense.

In essence, they fit the reduced model, then permuted residuals from that are added to the fitted values for the reduced model to produce a new "resampled" Y* which is then regressed on all the predictors to obtain a "randomization-like" distribution for the test statistic of interest.

You might like to compare this with another resampling technique, the bootstrap which involves resampling with replacement (while randomization tests resample without replacement), one version of that for the regression situation, the residual bootstrap operates in a similar fashion. (If you seek a reference, I'd suggest a recent printing of the book by Davison and Hinkley [2])

[1] Freedman D. and Lane, D. (1983),
A Nonstochastic Interpretation of Reported Significance Levels,
Journal of Business & Economic Statistics, Vol. 1, No. 4 (Oct.), pp. 292-298

[2] A. C. Davison. & D. V. Hinkley (1997),
Bootstrap Methods and their Application,
Cambridge Series in Statistical and Probabilistic Mathematics,
Cambridge University Press

gung - Reinstate Monica
  • 145,122
Glen_b
  • 282,281
  • This is a really informative answer, "parallels" confused me too. – NULL Oct 20 '17 at 14:33
  • What if the dependent variable or basically label is binary? Should one follow the same approach or label permutation would be ok? this approach is also discussed here – NULL Oct 20 '17 at 16:14
  • 1
    @NULL Thanks for the edit. I don't quite follow the setup of your question. Note that permutation of the observations which might swap $(y_i,\mathbf{x}_i)$ with $(y_j,\mathbf{x}_j)$ (this is what would normally be called 'permuting the labels' in relation to permutation tests - you could swap the i and j labels) would achieve nothing - you're just changing the order of the values. On the other hand, you can't just permute the responses (swap y's across the x's so you would get say $(y_j,\mathbf{x}_i)$) since the y's are not exchangeable -- ...ctd – Glen_b Oct 20 '17 at 23:21
  • 1
    ctd... they have different conditional distributions under the null, unless there's only one $x$ (the one you're omitting to do the test). While it's often possible to construct something that's more-or-less close to exchangeable with a general linear model you don't usually have the same thing with generalized linear models; the binomial (0/1 case) would be a particularly difficult one to deal with that way. (With the gamma glm, I might argue you could try to exchange Anscombe residuals perhaps or deviance residuals, maybe. I don't think that would be feasible with a 0/1 response) – Glen_b Oct 20 '17 at 23:33