1

We want to know whether two sample sets {x} and {y} were drawn from the same distribution. The null hypothesis $H_0$ is that they are. As statisticians we test the hypothesis by calculating the p-value: the probability of observing those sample data if the null hypothesis is correct.

As statisticians with computers we can avoid thinking too hard about the characteristics of their distribution and just run a permutation test on whatever statistic we care about. Well, we said earlier that we care about whether they are different. So I guess we have to come up with some MLE $\hat{\theta}$, and then look at the difference: $\hat{\theta_x}-\hat{\theta_y}$, right?

But we've been reading all about Likelihood Ratio Tests, which have some nice features, and a difference is just a ratio in log space, so: are there advantages to running the permutation test on the ratio $\frac{\hat{\theta_x}}{\hat{\theta_y}}$ instead of the difference? IIUC, using the ratio gets us into a situation, via Wilks' theorem, where we approximate the $\chi^2$ distribution. But we already decided our computer can do a permutation test – potentially even exhaustively for small sample sizes – and isn't that preferable in practice to a closed-form solution that is only asymptoticly correct?

Or maybe we pick up some power if we mix LRT with the permutation concept and look at $\frac{\hat{\theta_{xy}}}{\hat{\theta_x}\hat{\theta_y}}$?

feetwet
  • 1,108
  • 1
    The likelihood ratio is the ratio of the likelihoods, which is not usually the ratio of the parameters. I'm sorry to say there is a lot else that isn't correct in your question statement, but that's probably the most obvious error. – jbowman Dec 01 '23 at 05:15
  • @jbowman The question includes statements of my current understanding, which I suspect contains errors and is missing important qualifications. Any explanations of things that I have wrong (and why) could be helpful answers. – feetwet Dec 01 '23 at 13:56

1 Answers1

2

There are hidden problems with permutation tests, the first and foremost being their reliance on choosing the “correct” statistic to permute. With heavy tails in the data distribution, the sample mean has problems with overly influential observations, and permuting the data doesn’t fix that. Using a likelihood ratio $\chi^2$ test or a Bayesian regression model implies that you have thought about the choice of the raw data model, and such thinking may involve transforming Y in a certain way. For example if Y has a log-normal distribution, one computes the mean and SD after taking log(Y), and the mean of Y on the original scale is estimated by anti-logging a function of the estimated mean and SDs of the logs.

Frank Harrell
  • 91,879
  • 6
  • 178
  • 397