General guidelines on how to derive a hypothesis statistical test?

Question

In general, the process of hypothesis testing can be divided in 4 steps:

1. Formulate the practical problem in terms of hypotheses.
2. Calculate a statistic $T$, a function purely of the data. All good test statistics should have two properties: (a) they should tend to behave differently when $H_0$ is true from when $H_1$ is true; and (b) their probability distribution should be calculable under the assumption that $H_0$ is true.
3. Choose a critical region. We must be able to decide on the kind of values of $T$ which will most strongly point to $H_1$ being true rather than $H_0$ being true.
4. Decide the size of the critical region. This involves specifying how great a risk we are prepared to run of coming to an incorrect conclusion. We define the significance level or size of the test, which we denote by $\alpha$, as the risk we are prepared to take in rejecting $H_0$ when it is in fact true.

It seems the most creative step, the one that really sets a specific test apart from others is the choice of the statistic $T$. Hence, my question is: How did authors of statistical hypothesis tests come up with their statistics?

Given a specific problem, is it always obvious what the ideal (if this is definable on objective grounds at all) statistic ought to be? It seems those two requirements listed in step 2 above are two broad and many different statistics could be devised to test the same hypotheses. For instance, would it have not been a different alternative test to the t-test based on medians or other statistic...?

Answer 1

How did authors of statistical hypothesis tests come up with their statistics?

There are numerous ways to identify test statistics, depending on circumstances. It's important to try to identify the alternatives you see as important to pick up and try to get some power against those, under some plausible set of assumptions.

If you have a hypothesis relating to population means (in fact, let's make it simple and consider a one-sample test), for example, a statistic based on the sample mean would seem like an obvious choice for a statistic, since it will tend to behave differently under the null and the alternative. However (for example), if you're looking at shift-alternatives for a Laplace / double-exponential family ($\text{DExp}(\mu,\tau)$), something based on the sample median would be a better choice for a test of a shift in mean than something based on the sample mean.

If you have a specific parametric model (based on some particular distribution-family), it's common to at least consider a likelihood ratio test, since they have a number of attractive properties for large samples.

In many situations where you're trying to design a test from scratch, a test statistic will be based on a pivotal quantity. The test statistic in a one-sample t-test (as well as with many other tests you may have seen before) is a pivotal quantity.

Given a specific problem, is it always obvious what the ideal (if this is definable on objective grounds at all) statistic ought to be?

Not at all. Consider a test of general normality against an ominibus alternative, for example. There are many ways to measure deviation from normality (dozens of such tests have been proposed), and at typical sample sizes, none of them is most powerful against every alternative.

In trying to design a test for a situation like that, a certain amount of creativity is called for in coming up with a choice that will have good power against the kinds of alternatives you're most interested in picking up.

It seems those two requirements listed in step 2 above are too broad and many different statistics could be devised to test the same hypotheses.

Indeed. If you make some parametric assumption (assume the data are drawn from some distribution family and then make your hypothesis relate to one or more parameters of it) then there might be a best-possible test for all such situations (specifically, a uniformly most powerful test), but even then if your parametric assumption is more like a rough guess, then a desire for some robustness to that assumption may change things quite a bit.

For example (again, taking a one sample test of location shift to be simple), if I am sampling from a normal population then a t-test will be best. But let's say I think that it may not be exactly normal and on top of that there might be a small amount of contamination by some other process with a moderately heavy-tail, then something more robust (perhaps even a rank based alternative like the signed rank test) may tend to perform better across a variety of such situations.

Answer 2

A useful test statistic is one whose distribution depends on the parameter of interest and no other part of the statistical model. That way its distribution under the null hypothesis (i.e., when the parameter of interest has the value specified by the null hypothesis) can be fully specified. An ideal test statistic adds to that the property of having a distribution that is strongly dependent on the parameter of interest so that the resulting test has good power.

Consider Student's t-test. It was developed as a significance test (see What is the difference between "testing of hypothesis" and "test of significance"?) for small sample means. The difficulty that Gossett faced was that the distribution of the mean of a small sample from a normal population depends on the parameter of interest, $\mu$, but also a 'nuisance parameter', the standard deviation of the population, $\sigma$. The small sample condition meant that the standard deviation estimated from the sample, $s$, is not an adequate estimate of $\sigma$. To solve the problem Gossett devised the test statistic $t=\sqrt{n}\times \bar{x}/s$ which is dependent on only the data and that has a defined distribution for any given sample size, $n$. Importantly, that distribution is entirely unaffected by $\sigma$. (Actually, that form of the test statistic was a revision by Fisher, if I remember correctly.)

Nowadays it is not always easy to see the genius of Gossett's solution, particularly as his t-statistic looks almost identical to the z-statistic for a normal distribution with known variance (just substitute $\sigma$ for $s$). The hard part was determining the nature of the distribution of the test statistic. Proof that Gossett's distribution was correct didn't come until a later paper by Fisher.

In many cases statistical tests are devised by finding test statistics that take a distribution that can be proved to approximate known distributions under assumptions that are acceptable. Many tests are based on approximations to the chi-squared distribution, for example.