Arbitrariness in statistical tests

Question

I am new to statistics and find the following procedure unnatural and arbitrary. Could someone point out what I miss, and how I should be thinking?

Assumptions

I have two vectors of real numbers $x_{i}$ and $x'_{i}$ with length $N$, and I assume that they are respectively sampled from one-dimensional normal distributions $N(0,\sigma^2)$ and $N(0,\sigma'^2)$. The goal is to tell how different $\sigma$ and $\sigma'$ are. A standard approach seems to be the F-test, described as follows:

F-test

Define the statistics $T(\{X_i\}, \{X'_i\})$ to be the ratio of the sample variances: $$\frac{\frac{1}{N}\sum_i X_i^2}{\frac{1}{N}\sum_i X_i'^2}.$$ If $\sigma$ and $\sigma'$ are close, the statistics $T$ should be close to $1$. With the hypotheses

H0: \sigma  = \sigma'
H1: \sigma != \sigma'

we assume that H0 is true. Then the cdf of $T$ is the regularized imcomplete beta function; lets just call it $F = F_{N,N}$. Now, we use the sampled statistics $t = T(\{x_i\}, \{x'_i\})$ (which is a number, assumed to be larger than $1$) to construct the p-value as the probability of the "extremes":

$$p = P(T > t) + P(T < \frac{1}{t})$$

Since $t$ is just a number, we can use $F$ to compute such $p$ given the observed data $x_i$. The idea is that, the more $t$ deviates from $1$ (the expected answer under H0), the smaller $p$ is. Therefore, if $p$ is small enough (normal practice $< 0.05$) we can reject H0 and accept H1.

Questions

1. I get that $T$ is a good enough statistics for this task. But is it universal in any sense? Can't one pick any other $T$ to fulfill the job? For example, $$\tilde{T} = \frac{1}{N}\sum_i X_i^2 - \frac{1}{N}\sum_i X_i'^2.$$ This is just an example: my point is not to argue this as an alternative, but to stress the arbitrariness.

2. Why should we construct $p$ in that way, while there are many other ways to "measure" how far $t$ deviates from $1$. For example, $$\tilde{p} = P(T > t) + P(T < 2-t).$$ I get that its more natural to use multiplicative scale here, but it still feels arbitrary. Again, this is just an example.

3. My true incentive is really to tell how much the $\sigma$'s differ. But now the test only allow me to tell if they are different, even if they only differ by $10^{-9}$.. How can one build a hypothesis test in order to quantify such difference? For example,

   H0: |\sigma - \sigma'| <  10
   H1: |\sigma - \sigma'| >= 10
   

   Can one even build a test where $10$ is abstracted?

Conclusion

I believe I have pointed out some arbitrariness in the F-test. However, every statistical test seems to be arbitrary in the same way. A different test statistics and a different way to construct $p$ will change the final result. So how should I understand our scientific results nowadays?

Answer 1

You can feel free to construct a test statistic this way. An advantage, however, of using a ratio is that there is a sense in which that is more natural for a measure of spread: “the data are twice as spread out as they used to be” might mean more than “the data are one unit if spread higher than they used to be”.

3)

This is related to something called equivalence testing. With a standard F-test, you might want to test if the variances are within a multiplicative factor of $2$ of each other, and as long as either variance isn’t more than double the other, the variances are “practically” the same. You can do this with an F-test for a multiplicative factor. For an additive comparison, you might have to use some kind of simulation, though it should be do-able.

The easiest type of equivalence test is called TOST: two one-sided tests. You also might be interested in calculating a confidence interval for the ratio of (or difference between) your two variances.

Finally, note that the F-test is quite sensitive to violations of the assumption of normality. If you’re really convinced that you have normal distributions, great! If you think you can rely on some kind of argument based on the central limit theorem like g’night for a t-test of means, however, that is a mistake. JBStatistics has a YouTube video about this.