Significance test for large sample sizes

Question

This is more of a theoretical question. Super large sample sizes will almost always show a significance when a $\chi^2$ test is done. Is there any other statistical test of significance (an alternative to $\chi^2$) which is good for testing independence when the sample size is very large?

This is the context of my problem: I have 2 large datasets of phrases. Set1 corresponds to the Google n-grams set and set2 is a smaller set corresponding to the phrases found in one single website. Now consider a phrase: say ('Technology') found in Set2. I want to test and see if this phrase is specific to this website (it could be if it is a Technology website) or if it is a general phrase. So I am performing a $\chi^2$ test between the frequency of phrases in the two sets as follows:

                        Set1                        Set2
Not_Technology   (set1) 2,674,797,869,255    (set2) 46,168,477.00 
Technology       (set1) 1710231              (set2) 1991

I understand that this might not be the best method to test whether a phrase is a general phrase or not, so if you have any suggestions or criticisms I am happy to listen to them.

Answer 1

The test is doing what it should be doing. You ask it if two quantities are equal, in the case of the original question, if zero is equal to some measure of independence that is zero when the distributions are independent (e.g., mutual information). Since the test has considerable sensitivity, due to the large sample size, the test correctly tells you that the two quantities are not equal. This is a design feature, not a bug, of hypothesis testing that is related to consistency (power converges to $1$ as the sample size increases).

If you remember the Princess and the Pea fairy tale, you may recall that, no matter how trivial we might perceive a pea under the mattress, the princess was correct about there being a pea. If you want to assert that a pea under the mattress does not matter to you, that's fine, but it is a mistake to call the princess incorrect for noticing the pea when the pea was indeed under the mattress.

Because of the consistency of most tests are the frequent availability of large amounts of data, hypothesis tests certainly can find differences that, while they are there, are not important or interesting, much like most people would not care if there is a pea under the mattress. This gets into the effect size and what kind of effect size is interesting. While statistics can (and has) come up with interesting ways to quantify effect sizes, determining an interesting effect size mostly falls outside of the realm of statistics and in the domains to which statistics is applied (the experts in medicine decide that for COVID studies, the experts in economics decide that for unemployment studies, etc).

Once you have an effect size of interest, there are a number of statistics tricks related to it. First is that investigators can calculate the sample size required for detecting such an effect size with a certain power and $\alpha$-level. This is not so important for the situation where you already have a ton of data, but it is worth a mention. Examples of this are available in the pwr package in R.

A second trick is equivalence testing, the easiest example of which to understand is two one-sided tests: TOST. Briefly, TOST does two hypothesis tests in order to bound our estimate of the true effect, rejecting that the true effect is too high or too low.

A third trick is interval estimation. A frequentist might calculate a confidence interval to put bounds on the effect size, and a large sample size would lead to a relatively narrow confidence interval and correspondingly high precision in the estimate. A Bayesian might calculate a credible interval for the same purpose. All else equal, the credible interval should be narrower for a larger sample size, with a large sample resulting in a tight estimate of the true effect. Whether you go frequentist or Bayesian, a tight estimate and high precision sounds desirable. Once you have a range of plausible parameter values given by the interval estimate, you can analyze if any of those have any practical significance by comparing to your effect size.

Depending on what you want to do, one of these three might be reasonable for handling situations where large data sets are available. However, they all need some declaration of an effect size of interest!

Answer 2

This is a general phenomenon in hypothesis testing for a point-null hypothesis

What you are dealing with here is a much larger issue in hypothesis testing than just the chi-squared test of independence. This is a phenomenon that arises in classical hypothesis testing whenever you are testing a point-null hypothesis (i.e., a null hypothesis that stipulates a single point for an unknown parameter). In such cases, the null hypothesis is a choice of a single parameter value, usually over an uncountable set of possible values. That is an extremely specific null hypothesis.

To understand the phenomenon you are referring to, let's have a look at the properties of a consistent hypothesis test. Hypothesis tests are designed to test the specified null hypothesis and reject it (in favour of a specified alternative) if the evidence falsifies the null hypothesis. (More information on the mathematical structure of a hypothesis test is avaialble in this related answer.) Suppose you have data $\mathbf{x}_n$ and an unknown parameter $\theta \in \Theta$ and you pick a null hypothesis space $\Theta_0 \subset \Theta$. Let $\alpha$ denote the significance level for the test and let $\beta_n$ denote the resulting power function. A consistent hypothesis test will have the following limiting property for its power function:

$$\lim_{n \rightarrow \infty} \beta_n(\theta) = 1 \quad \quad \quad \quad \quad \text{for all }\theta \in \Theta-\Theta_0 \text{ and } 0< \alpha <1,$$

which implies the following limiting property for its p-value function:

$$\underset{n \rightarrow \infty}{\text{plim}} \ p(\mathbf{x}_n) = 0 \quad \quad \quad \quad \quad \text{for all }\theta \in \Theta-\Theta_0. \quad \quad \quad \quad \quad \quad \quad $$

---

Consistency under the point-null hypothesis: Hypothesis tests are designed to test the truth or falsity of the hypotheses you actually give them, so if you use an extremely specific null hypothesis, and that hypothesis is even slightly false, the test is designed to correctly infer that the null hypothesis is false. In particular, if the point-null value is $\theta_0$ then for any parameter value $\theta \neq \theta_0$ you will have $\text{plim}_{n \rightarrow \infty} p(\mathbf{x}_n) = 0$ (i.e., the p-value will converge stochastically to zero).

One of the problems that arises in hypothesis testing occurs when we set a point-null hypothesis in a circumstance where that specific hypothesis is almost certainly false, but what we really want to know is something a bit broader --- e.g., whether the specified point-null is almost correct. A common case occurs when the parameter can be considered to be a continuous random variable, such that it will be equal to the stipulated point-null value with zero probability.$^\dagger$ In this case, the null hypothesis is false with probability one and so with a large amount of data the test gives us a tiny p-value which tells us that the null is false. Some users view this as a deficiency of the hypothesis test, but it is actually a case where the test is doing exactly what you ask it to. By specifying a point-null hypothesis you are asking the test to be very specific about the null hypothesis under consideration, and the test is complying with this instruction.

So, what can you do to deal with this "problem". Firstly, you ought to recognise that you need to test the hypothesis you are actually interested in, not a hypothesis that is mathematically close to this but much more specific. Typically you can do this by setting some "tolerance" $\epsilon>0$ on your stipulated point-null value and testing the composite null hypothesis $\theta_0 - \epsilon \leqslant \theta \leqslant \theta_0 + \epsilon$. You can view the tolerance value as a measure of "practical significance", meaning that if the true parameter value is within the stipulated tolerance of the point-null value then it is "practically" equivalent to the point-null value. In this manner you can separate "statistical significance" from "practical significance" and ensure that the consistency property of the hypothesis test does not lead the p-value to converge to zero in cases where you don't want it to.

Implementation of a "tolerance" in the null hypothesis for the chi-squared test of independence is quite complicated and so outside the scope of the present post (but feel free to ask a separate question for how to do this). In general you can alter existing tests to include a tolerance on a point-null hypothesis but you need to re-derive the test as a composite test to determine how the composite hypothesis affects the p-value function. This is a complicated exercise in general, but it can be automated into customised p-value functions once derived.

---

$^\dagger$ You will sometimes see statistical commentators make a broader assertion that a point-null hypothesis is always false. That is not true --- a point-null hypothesis can be true. Moreover, even in the case where the parameter is viewed as a random variable, it can be equal to a specific value with positive probability. It is only if we are willing to stipulate that the parameter is a continuous random variable that it has probability zero of being equal to any specific value.

Answer 3

Take a look at this paper by the late Jack Good: http://fitelson.org/probability/good_bnbc.pdf ; At section 4.3, his "Bayes/Non-Bayes Compromise" leads to the definition of a "standardized" p-value which tries to address the "Huge $n$ $\Rightarrow$ Highly Probable to Reject Null, Whatever Data" effect.

Answer 4

If your sample is large enough then it seems to me that a statistical test is not needed. You have characterised the effect. Is the effect that you have characterised large enough to be interesting? If so, then make a reasoned and principled argument about the observations without recourse to a testing procedure.

Answer 5

Congratulations! You have a big enough sample size that you don't need to bother with significance testing! So don't worry about it. Now you just need to decide if the effect you see is "big enough" to care about, which is an entirely different problem that has nothing to do with significance testing.

A statistical significance test, like a Chi square test, is attempting to answer a very specific problem: "how likely is it that a difference I observe in a random sample is just an artifact of sampling error (the error that arises when we try to make generalizations about a population using only a random sample of that population)?" That's it. The fact that a test is significant doesn't tell us anything about whether the effect is "big" or "meaningful" in some substantive sense, or even that it's actually "real" (it might be due to some measurement error, or confounding with other variables).

Now, as sample size increases the likelihood of an observed difference of a given size being an artifact of sampling error goes down, and so significance tests will tend to be significant basically all of the time. This is just because they problem they are trying to help you solve has already been (largely) solved by the large sample size.

So, in your case sampling error is not a particularly big problem, so a significance test is not very helpful. Rather, what you need to decide is if the relationship you are looking is "big enough" or not. That's not a question that can be answered with a statistical test. You need to use your knowledge of the subject matter to decide if the relationship is large enough to "make a difference in the real world." No statistical test can answer that question for you, and neither can anyone here, unless they also happen to know a lot about Google n-grams, and the specific research question you are asking.

Answer 6

When we do sample size determination for clinical trials we define a clinically significant (or clinically important) difference. That is a difference that is large enough to be worth detecting. The definition is given by the clinician. It is not a statistical issue. It depends on the clinical problem and requires a clinical judgement. Once the clinician has decided on that we pick the smallest sample size required to have high power (80% or more) for detecting a difference that large.

In your case where you already have millions of samples what you can do is rephrase the question. Instead of the standard null hypothesis that the difference is 0 which you reject if you can determine that it is any size different from 0, define a delta that represents what you think is an important distance. Then you reject the null hpyothesis only if the test indicates that the difference is greater than delta.

Answer 7

The problem here is that it is bad statistical practice to propose a statistical test as part of an analysis plan without any knowledge of the sample size. If you did know the sample size, you should adjust the alpha accordingly. Ideally, a highly calibrated test such as the one you get with having N very, very large should balance the false positive error rate with the power - not in any objective sense, but there's no reason to set the usual (arbitrary) 0.05 when the sample size is so large a "significant" result is basically meaningless.

The only way to know this is by doing simulations and understanding how the test statistic behaves when the null is true. There's no reason you can't set alpha = 0.0000001 and look to the one-minus-one-millionth quantile to define the critical value of the one-sided test, which may be a suitably large quantity.

Answer 8

This is my current recommendation:

Statistical Tests (Hypothesis Testing)

A statistical significance test, like a Chi square test, is attempting to answer a very specific problem: "how likely is it that a difference I observe in a random sample is just an artifact of sampling error (the error that arises when we try to make generalizations about a population using only a random sample of that population)?"

Small Sample Size

Summary

What is a small sample size? When power is not too high (e.g. 0.9999) and p-value/CIs are not to small, usually n<500 or 300.

Summary what to do for small sample size Hypothesis Testing: - p-value: stat test (e.g. t-test) with p-value & significance level - effect size: report the effect size (e.g. Cohen's d), see if it falls in the common ~0.2 (small), 0.5 (medium), 0.8 (large)
& compare it to eps/pooled_std(group1, group2) - CI: CI's, do they intersect given the epsilon that matters for your application? - Power/sample size: making an estimate of your std (or preliminary data), get Power of your test with a given sample size or compute the sample size you need to achieve good power.

ref: - also fantastic reference: https://stats.stackexchange.com/a/602978/28986

Large Sample Size

Same as previous comment but n>500 good rule of thumb.

Summary what to do for large sample size Hypothesis Testing: - CI: CI's and use the epsilon valid in your applicaiton - Effect size: report effect size, see if it falls in the common ~0.2 (small), 0.5 (medium), 0.8 (large) & compare it to eps/pooled_std(group1, group2) - LRT: todo - eps != 0 p-value: todo

Hands on example

See my_test_using_stds_from_real_expts_() function in effect_size.py.

Todo later (for large sample size)

• LRT (theory & python) mainly for large sample size
• hypothesis testing with non-zero epsilon (theory & python) mainly for large sample size

ref: - Fantastic reference: Significance test for large sample sizes