T-test on entire set of data vs. 2 t-tests on 2 halves of the same set of data

Question

I have a set of data samples which have been gathered from an electronic system. The system is in one of two states, which is selected at random before the data sample is collected. For simplicity, let us say that each sample is a single value. I collect thousands (sometimes millions) of data points.

My null hypothesis is that the means of the two sets of samples are the same - which (in my actual case) implies that the data samples are not capable of telling me what state the system is in (although I note Nuclear Hoagie's comment.)

To test my hypothesis, I separate my samples out into 2 sets, based on the state, and then perform Welch's t-test between the two sample sets. If the absolute value of the t-statistic is > 4.5, I reject the null hypothesis (implying that my data samples can distinguish between the states of the system) with high confidence (if I have many thousands of samples, I believe my confidence is greater than 99.999%?). I am looking for tiny, but real, differences, so the usual 95% confidence is not sufficient.

So far, so (I think) standard.

Now, in this field of endeavour, there is a "standard way of working" which says that if you reject the null hypothesis, you should do another experiment, just the same, and check that you get the same result - to give stronger confidence.

This raises some questions (which may be very simple, but I am always surprised by how statistics questions go against my intuition! For context, I am not a statistician, I am an electronic engineer/computer scientist :):

• What is that new confidence? Is it as simple as $1-(0.00001^2)$?
• If I am going to run 2 identical experiments, how is that different to just running the first t-test with twice as much data?
• Can I get the same level of confidence on the "longer experiment" just by changing the threshold? How do I calculate what that threshold would be?

Answer 1

Difference in power

When you split a data set and perform two separate tests with a cutoff significance level of $\sqrt{\alpha}$, then the test will be less powerful.

The simulations below demonstrate this for the case of a z-test (it is a bit more difficult to compute for Welch's t-test but the principle is similar) when the significance level is $\alpha = 0.04$.

• Left/right panels

  On the left side the hypothesis test is based on the z-score of the combined data.

  On the right side the hypothesis test is based on the z-scores of the split data.

• Top/Bottom panels

  For the top panels the data is generated assuming that the null hypothesis is correct. This will give a uniform distribution of the p-values. We see that both regions will cover roughly 4% of the area (small discrepancies due to variation in the simulations).

  In the bottom panels the data is generated for when the alternative hypothesis is correct with an effect size of $E[z] = 0.5$. This will give a distribution that is not uniform. We can see that the two different tests will have different probabilities in rejecting the null hypothesis. The left side, the test for the grouped data, will be more powerfull (reject more often).

  The reason for the higher power is because the test on the left will cover the regions where the alternative hypothesis has the highest density. It is the area that covers 4% of the cube that contains the highest density of the cases given the alternative hypothesis (think of the Neyman Pearson lemma).

Why sacrifice power this way?

If I am going to run 2 identical experiments, how is that different to just running the first t-test with twice as much data?

As explained above a difference is the power of the test. But that is not beneficial. Why then follow this procedure?

The reason is because with the split testing, you are not always running that second experiment. That saves you time, money and energy. You can see it as an early stopping rule.

Also, there may be several practical issues like the ones mentioned by Lukas Lohse. The analysis above is a computation for significance tests assuming that testing procedure is exactly as described by the statistical model. Those assumptions might be wrong. (e.g. systematic bias is not captured by statistical variance in the sample) Performing an additional tests can be considered as a way to make the testing more robust. Ideally that second test is not identical, but is completely independent (to eliminate systematic bias).

Code for image

set.seed(1)
layout(matrix(1:4, 2, byrow = TRUE))
par(mar = c(4,4,4,2))
n = 5*10^4
alpha_t = 0.04
alpha_s = sqrt(alpha_t)
zt = qnorm(0.04)     # cutoff level both sets
p1 = seq(0,1,0.01)   # p-value first test
z1 = qnorm(p1)       # z-value first test
z2 = zt*sqrt(2)-z1   # z-value second test
p2 = pnorm(z2)       # p-value second test
simulations with null hypothesis true
simulations with null hypothesis false
for (i in 0:1) {
sim_z1 = rnorm(n, i-0.5)
  sim_z2 = rnorm(n, i-0.5)
  sim_p1 = pnorm(sim_z1)
  sim_p2 = pnorm(sim_z2)
  test_1 = pnorm(sim_z1+sim_z2, 0, sqrt(2)) < alpha_t
  test_2 = (sim_p1 < alpha_s) * (sim_p2 < alpha_s)
plot(-10,-10, xlim = c(0,1), ylim = c(0,1), xlab = "first p-value", ylab = "second p-value")
  title("test using single data set", line = 0.5, font.main = 1)
lines(p1,p2, lwd = 2)
  points(sim_p1, sim_p2, pch = 21, cex = 0.5,
         col = rgb(test_10.4+0.6, 0.6, 0.6, 0.02), 
         bg  = rgb(test_10.4+0.6, 0.6, 0.6, 0.02))
text(0.2,0.3, paste0(round(100*mean(test_1), 2), " %"), col = 2)
plot(-10,-10, xlim = c(0,1), ylim = c(0,1), xlab = "first p-value", ylab = "second p-value")
  title("test using split data sets", line = 0.5, font.main = 1)
lines(c(alpha_s,alpha_s,0),c(0,alpha_s,alpha_s), lwd = 2)
  points(sim_p1, sim_p2, pch = 21, cex = 0.5,
         col = rgb(test_20.4+0.6, 0.6, 0.6, 0.02), 
         bg  = rgb(test_20.4+0.6, 0.6, 0.6, 0.02))
text(0.2,0.3, paste0(round(100*mean(test_2), 2), " %"), col = 2)
if (i == 0) {
    mtext("simulation of p-values when null hypothesis is true", side = 3, line = -2, outer = TRUE, font = 2)
  }
  if (i == 1) {
    mtext("simulation of p-values when alternative hypothesis is true (d = 0.5)", side = 3, line = -23, outer = TRUE, font = 2)
  }
}

Answer 2

There is a lot to say here and I think it would helpful for frankly everyone to read the Reckless Guide to P-values from Michael J. Lew, in which he explains the nature of significance and hypothesis testing very well: https://link.springer.com/content/pdf/10.1007/164_2019_286.pdf?pdf=inline%20link

Also I think that you are confusing significance and confidence as statistics terminology: $\alpha$ significance is a property of data $x$ relative to a hypothesis $H$, namely $P(X \text{ "more extreme than" }x|H) < \alpha$, while $1-\alpha$ confidence is the property of a region (e.g. an interval) of hypothetical parameter values, such that each value outside of it would fail an $\alpha$ significance test given your data $x$.

Simple math answers to your questions

1. What is that new confidence? Is it as simple as $1−(0.00001^2)$?

Under the Null-hypothesis the two samples are independent and so the probabilities multiply, making it a legitimate $0.00001^2$ significance test.

2. If I am going to run 2 identical experiments, how is that different to just running the first t-test with twice as much data?

I'll have a lot more to say about this in the 2nd part of the answer but for now: Given the large sample sizes and assuming constant variance we take the t-statistics in the two datasets as $T_1, T_2\sim\mathcal N(0,1)$ under the Null-hypothesis. Now in the combined data the standard error is smaller by a factor of $\sqrt 2$ while the difference of means is just the average of the differences of means in the two subsets. So $$ T_{combined} = \frac{T_1 + T_2}{2}\cdot\sqrt{2} $$

Now we have two (two sided) $\alpha^2$ significance tests:

1. $|T_1| > \Phi^{-1}(1-\alpha/2) \text{ and }|T_2| > \Phi^{-1}(1-\alpha/2)$

2. $|T_{combined}| > \Phi^{-1}(1-\alpha^2/2)$.

Plugging in $\alpha = 10^{-5}$ we find that $T_1, T_2$ have to be bigger than $4.42$ for 1. and the mean of $T_1, T_2$ has to be bigger than $4.57$ to pass 2. So both being $4.5$ fulfills 1 but not 2, while one being $4$ the other $6$ works only for 2. The 2nd Test would have higher power under the assumptions of a t-test making it the standard.

3. Can I get the same level of confidence on the "longer experiment" just by changing the threshold? How do I calculate what that threshold would be?

Not really a math-question, but in the Fisher-approach you would report your p-value as precise as would make sense, while in the Neymann-Pearson-approach to hypothesis testing it is completely unacceptable to change the significance levels (really the critical values) after the fact. Again i implore you to read the link posted at the top.

Serious considerations

From what you are writing the probability of your results coming from purely random noise is basically zero, however there are an awful lot things that are neither purely random noise nor interesting scientific results, which often means that a 2nd round of data collection ends up very much non identical to the 1st one (https://en.wikipedia.org/wiki/Replication_crisis).

1. Measurement error: A loose plug disproved the general theory of relativity with like $|t| > 6$ in 2011: https://en.wikipedia.org/wiki/Faster-than-light_neutrino_anomaly#Measurement_errors
2. Unmeasured confounding: Maybe your circuit picks up thunder storms and there were more thunder storms while you were measuring state A then while measuring state B.
3. Inconsistent effect: Maybe the difference between states is that A picks up on thunder storms but B doesn't. You won't measure a difference when the weather is sunny.
4. Issues in the statistical analysis like:

• implicit multiple comparison
• reinforcing random patterns during data cleaning
• violations of assumptions, e.g. non-constant variance between states
• ...

Repeating your experiment is much more about reducing theses sources of error than about lowering the significance level, which really only deals with purely random noise.