What are the benefits of classical ANOVA F compared to Welch's F?

Question

Is there a reason to report classical ANOVA F values (Fisher's F), instead of Welch's F-tests?

I know Welch's F test is corrected, so that it's robust against heterogeneity of variance. This leaves me wondering: Is there any reason why studies don't regularly only report Welch's F?

score 10 · Answer 1 · answered Jan 17 '24 at 19:17

10

Several points:

The first thing is for the significance level calculations to be valid the assumption would be about the situation when $H_0$ is true. It would be possible for the distributions to have the same (or at least very similar) spread under $H_0$ but to differ as the means move (e.g. in a situation where the spread increased when the mean increased, as might be quite plausible with a non-negative response where if $H_0$ is true the group-factor might be expected to do nothing at all - such as a completely ineffectual treatment, perhaps - so that all groups would effectively be a single population).

Which is to say, the incidental appearance of the sample data may not necessarily be particularly relevant to the consideration of the assumption required to obtain correct or almost-correct significance levels and p-values (which related to the case where $H_0$ holds, which will usually not be the case in the observed data, even if you fail to reject the null).
The significance level in one way ANOVA is highly robust to the variances being different under $H_0$ in the case where the sample sizes are equal.
Those things aside, there's usually very little lost when applying the Welch-Satterthwaite approximation even when the population variances are exactly equal. Which is to say, it is usually a perfectly reasonable default if you are not confident whether assuming equal variances under the null would make sense in your situation.

In summary: sometimes it can make sense to use the equal-variance version even when it might look like you should not (not that I am advocating choosing your test on the basis of what you happen to see in the data you want to use the test on). On the other hand, there's usually little to lose by using Welch-Satterthwaite ANOVA as a matter of course.

answered Jan 17 '24 at 19:17

Glen_b

282,281

Thanks. However, if I get you right, in point 1) you're saying it's not often classical ANOVA is worse than Welch's F, which doesn't seem like an argument to use it to me. Point 3) sounds like precisely what I'm interested in - is there really any information lost, when we only report Welch's F? I.e. could theoretically Welch's F come out significant and Fisher's F non-significant, so that their contrast would indicate something meaningful? – foggy Jan 18 '24 at 13:45
I don't quite agree with your characterization of what I said in 1. Something that happens less often than many people think is not necessarily all that rare. It's a matter of understanding the circumstances where it works just fine, and thinking about whether that might apply. In some application areas it will be happening a lot and in others, perhaps not so much. 2. Be careful not to conflate performance across data sets (any discussion of significance level or power) with something specific to a single data set (one being significant, the other not). That can happen both ways. ... ctd

Glen_b

Jan 18 '24 at 16:19

1

... I did give both specific and general advice. One downside of the Welch that I didn't make explicit is that it's no longer exact even when the assumption all hold exactly. Whether you think that's an issue is another matter. I generally don't but people that get very antsy about not exceeding nominal significance levels (I see it being a very strong concern in specific circumstances and then almost completely ignored in others), might need to worry about it in this situation if they're going to behave consistently. I'll consider edits if I can see how to be both clear & fairly accurate – Glen_b Jan 18 '24 at 16:24

So if I get you right, you're saying that indeed in some cases Welch's F could come out significant and Fisher's F non-significant and it would be a reason to dismiss the hypothesis, because Welch's F is less accurate? – foggy Jan 20 '24 at 12:19

No, again I'm talking about properties across data sets (population-level considerations of the test properties like significance level) while you're focusing on properties (/happenstance) of one data set. – Glen_b Jan 21 '24 at 02:22

score 5 · Answer 2 · answered Jan 18 '24 at 09:17

A good reference that addresses your question, citing appropriate literature, is Delacre, M., et al. (2019). Taking Parametric Assumptions Seriously: Arguments for the Use of Welch’s F-test instead of the Classical F-test in One-Way ANOVA. International Review of Social Psychology, 32(1): 13, 1–12. DOI: https://doi.org/10.5334/irsp.198. (This reference may also be useful to future readers who are looking for the definition of Welch’s F-test.)

Taking literally from this paper:

Violation of Homogeneity of Variances Assumption Regarding the Type I error rate, the F-test is sensitive to unequal variances. When there are more than two groups, the F-test becomes more liberal, meaning that the Type I error rate is larger than the nominal alpha level, even when sample sizes are equal across groups. Moreover, when sample sizes are unequal, there is a strong effect of the sample size and variance pairing. In case of a positive pairing (i.e. the group with the larger sample size also has the larger variance), the test is too conservative, meaning that the Type I error rate of the test is lower than the nominal alpha level, whereas in case of a negative pairing (i.e. the group with the larger sample size has the smaller variance), the test is too liberal

Regarding the Type II error rate, there is a small impact of unequal variances when sample sizes are equal, but there is a strong effect of the sample size and variance pairing. In case of a positive pairing, the Type II error rate increases (i.e. the power decreases), and in case of a negative pairing, the Type II error decreases (i.e. the power increases).

Simulation studies performed in the aforementioned paper show that Welch's F-test appears to perform better than alternatives. Lastly, you may find helpful this interesting answer by gung - Reinstate Monica.

This is the information that lead to me ask the question, why we don't use Welch's F for all cases in the first place — foggy, Jan 18 '24 at 13:39
@foggy It would probably have been useful to mention that in your question; if what I've seen in the parts of the paper I can readily access online are any indication there are some issues with some arguments the paper, albeit I still broadly agree with the conclusion. — Glen_b, Jan 18 '24 at 16:35

score 2 · Answer 3 · answered Jan 17 '24 at 18:24

Yes, the classical ANOVA F-test is inappropriate if the variances, $\sigma^2$, of $Y$ across the groups being tested are unequal (heteroscedasticity). Welch ANOVA should always be used for unequal variances. So your question concerning "reason" is based on not violating an assumption regarding equal or unequal variances. Classical ANOVA assumes equal variances, Welch ANOVA does not.

What are the benefits of classical ANOVA F compared to Welch's F?

3 Answers3