How to understand that there are n - 1 degrees of freedom in calculating sample variance?

Question

According to Wiki: In statistics, the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary.[1] Estimates of statistical parameters can be based upon different amounts of information or data. The number of independent pieces of information that go into the estimate of a parameter is called the degrees of freedom. In general, the degrees of freedom of an estimate of a parameter are equal to the number of independent scores that go into the estimate minus the number of parameters used as intermediate steps in the estimation of the parameter itself. For example, if the variance is to be estimated from a random sample of independent scores, then the degrees of freedom is equal to the number of independent scores (N) minus the number of parameters estimated as intermediate steps (one, namely, the sample mean) and is therefore equal to N-1.[2]

Reference：https://en.wikipedia.org/wiki/Degrees_of_freedom_(statistics)

I am confused that because we know the sample mean which means that $n$ observations are already known so why we can let $n-1$ observations be free to vary? I checked some videos and blogs: most of them explain that by using a example that if we have a sample of 3 and know the sample mean two of them are free to vary and the third observation will be determined by sample mean. But if we already know the sample mean why two of them are free to vary? I think these three observations are fixed.

Answer 1

According to Wiki: In statistics, the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary.

I agree with you that this description is not very clear. Indeed, all $n$ of the variables are able to vary. The point is that they are constrained to a space that can be described by $n-p$ parameters.

Useful related questions

• Why are the residuals in $\mathbb{R}^{n-p}$?
• Obtaining the chi-squared test statistic via geometry
• Why divide by $n-2$ for residual standard errors

The following two images might be helpful:

Both images display the results of a low dimensional situation, with 3 variables such that they can plotted easily, but you may imagine how the logic is general for a higher number of variables.

illustration 1

Samples with $n=3$ observations $y_1$, $y_2$ and $y_3$ are fitted with a linear model of $p=2$ parameters.

The values $\hat{y}_i$ that the linear model can fit can be considered as a linear plane, the span of the vectors $x_1=[1,1,1]$ and $x_2=[1,2,3]$. The residual is the difference between the observation and the fitted point in the plane and this is always pointing in the same direction. That is, the residuals are a multiple of the vector $[1,-2,1]$ and can be described by a single value, the multiplication factor.

Illustration 2

This is an image from an early version of an answer here. The image relates to a chi-squared test for a table with 3 cells/values $x_A$, $x_B$ and $x_C$. Due to a constraint on their sum, they are fixed in a plane which you could see as a 2d distribution instead of a 3d distribution.

---

I just mean that becuase if we know the sample mean which reveals that the prior condition is that we know each value of the observation in that sample because we can calculate the sample mean only with knowledge of these values.

A sample is a fixed realisation from a distribution that allows some variation. Every sample will be different and each experiment you have a different residual. So for you specific observation the residual is fixed, but if you consider all potential outcomes then the residual is considered a random variable.

The number of degrees of freedom describes the dimension of the distribution of the residuals.

For example in the first image, points for different experiments/observations are plotted. In each case, the residual is always the value $a \cdot [1,-2,1]$, where $a$ can variate. It is a 1 dimensional space and 1 parameter can describe the distribution of the residuals.

Answer 2

I think PBulls and Sextus Empiricus gave a very clear, statistical answer already.

Maybe let me give 1 other, basic, non-statistical example:

Imagine you are a football manager and you have to put 11 players in a team where you have 11 fixed spots (1 for each player). So here: n = 11. Now you can put 10 players wherever you like (so you have 10 = 11-1 degrees of freedom), but the final player will be placed in the final open spot.

So back to your problem: The same happens with your sample: if you have a fixed mean (let's say $\mu$) from a sample of n = 4 observations, you can just randomly pick 3 number (3 = 4-1 degrees of freedom thus), but the final number should be in that way that your mean is still equal to the prespecified $\mu$ and therefore you are not free to choose this last number.

Now, in statistics, when you want to calculate some measure, like the mean, the degrees of freedom will tell you how many 'independent observations' you have that contribute to this estimate. In the example that you have 4 observations in total, you have 3 pieces of independent observations and you have 1 observation that is not free to vary anymore, but that is restricted by your other observations. So in that sense, you can say it's a way to quantify how many 'independent' data you have that is free to vary when calculating a statistic like the mean. Now it is hopefully clear that you of course first have your data and then calculate the mean, but this example shows how many of your observations are free to vary, when calculating this mean.

Closing example with numbers: let's say that you have a mean equal to 5 and a sample consisting of 4 observations (so n = 4). You can now ask the audience to shout 3 random numbers (these are you degrees of freedom, you can vary them as you like), and let's say they come up with 4, 5 and 10. Now the last number in your sample (let's call it X) is not free anymore, but should be such that $\frac{4 + 5 + 10 + X}{4} = 5$ (this is how you calculate the mean) and transforming this equation, you can find your last number: 1, which is now fixed, depending on your other 3 choosing values and the mean you want.