4

While reading this blog on f-statistic in ANOVA, I stumbled upon the formula:

The f-statistic is calculated like this: $$\frac{\text {variance between groups}}{\text {variance within groups}}$$

So, I needed to understand what exactly "variance between groups" and "variance within groups" mean.

Enter the F-test. We are going to state that if there is no difference in the means then the estimate of variance you get from the difference in group means should be the same as the estimate of the population variance you get within groups.$^1$

(emphasis added)

Well, after reading this, one can get an idea about what these terms really mean. Moreover, the formulas (unsourced) representing these terms are also given. Namely:$$\text{within group variance}=\frac{(n_1-1)s_1^2+(n_2-1)s_2^2+(n_3-1)s_3^2}{N-g}$$ where, $n_i$ is the number of elements in the $i^{th}$ group.
$s_i$ is the variance in the $i^{th}$ group.
$N$ is the total number of subjects.$^2$
and, $g$ is the number of groups. And, $$\text{between group variance}=\frac{\sum_{i=1}^g n_i(y_i-\bar y)^2}{g-1}$$
Where, $g$ is the total number of groups
$y_i$ is the group mean of the $i^{th}$ group.
And $\bar y$ is the overall mean.

But still, I do not clearly understand these terms. The main reason for this confusion may be due to these unsourced formulas.

NOTE:

  1. My main aim was to find how F-statistic compares two models in ANOVA.
  2. Secondly, I also wanted to know the reason for the fact that if the null hypotheses is true, then the"variance between groups" is equal to the "variance within groups".

Therefore I'm trying to understand the meaning of these terms.

$1$: The "no difference in the means" is the null hypotheses. The means are of three different groups.
$2$: Don't understand the "subjects" here.

Mooncrater
  • 787
  • 2
  • 9
  • 20

2 Answers2

2

The first term -- the within-group variance -- is simply a weighted average of the variances within each group (the $s^2_i$ values). If the population variances are the same (as is assumed in ordinary ANOVA) this is a natural way to estimate the common population variance.

(note that the term "subjects" just means the units you're taking the measurements on; whatever things had the values you collect)

If there were no differences in population mean, the sample means would vary (be different from each other) just due to randomness. The amount they would vary is proportional to how much variation there is in the population about the population mean -- i.e. to the common population variance.

Imagine for the moment all the groups are the same size. We could just treat the set of means for each group like a sample and calculate their variance.

That would be smaller (because the means are less variable than the data they came from), but in a specific way we can calculate. (The situation is similar when the sample sizes differ but it's a bit more complicated; nevertheless we can still get something that's expected to be proportional to the variance.

So we can actually scale the variation of the means from the overall average up to be an estimate $\sigma^2$ and if the population means are really the same it should be about the same size as the one from the within-group calculation, since they both estimate $\sigma^2$.

On the other hand, if you shifted population means so that the population means were very different (other things being the same), the denominator wouldn't change, but the numerator would become larger -- besides the part due to noise there would be another part due to differences in population means. So the numerator would then be larger than the denominator, and your F statistic would tend to be bigger than you'd anticipate when the null hypothesis was true.

My main aim was to find how F-statistic compares two models in ANOVA.

That sounds like you're talking about partial F's; the F in one-way ANOVA may not be particularly helpful for that.

Glen_b
  • 282,281
1

No matter the context, the F-test's null hypothesis is equal variance. In the case when you're comparing the "performance" of two candidate regression models, you should only use the F-statistic to test "nested" models (where the set of dependent variables of one candidate is entirely contained in the set of another that has more dependent variables). Only in this special case does testing for a significant difference in explained variance justify the decision to discard one model in favor of another.

With a little work, it can be shown that the F-statistic is also the degrees-of-freedom-corrected ratio of sums of squared residuals for each model, a much more intuitive set of terms to work with.

The reason for "between" and "within"-group nomenclature is a set of uses for the F-statistic from industrial engineering that has to do with measuring variation in manufactured parts. A key concern in these analyses is that the measurements (say, width at the base) of different kinds of parts should be more different than measurements taken on parts of the same kind. If this criterion is not met, the "gauge" or measurement device cannot be evaluated properly.

In more general contexts, the same math is interpreted as a test for homoskedasticity (constant variance).

grim_trigger
  • 11