6

Throughout my student life so far, I have always considered the mean squared error to be calculated by $ MSE=\frac{1}{n}\sum(Y_i-\hat{Y}_i)^2$. However I was looking at one of my statistics mod today and it was stated in the slide that

enter image description here

And that would mean that $ MSE=\frac{1}{n-2}\sum(Y_i-\hat{Y}_i)^2$ since $ SSE=\sum(Y_i-\hat{Y}_i)^2$.

Upon researching on this, I found this description on wikipedia:

mean squared error is sometimes used to refer to the unbiased estimate of error variance: the residual sum of squares divided by the number of degrees of freedom. This definition for a known, computed quantity differs from the above definition for the computed MSE of a predictor, in that a different denominator is used.

I would like to know if there is a correct definition or are the 2 MSEs here actually referring to completely different concepts? How do I go about understanding the reason for the difference?

tangolin
  • 177
  • 1
  • 6
  • 1
    These are estimators of the mean square error. Just to confuse things, the unbiased estimator, the maximum likelihood estimator and the estimator which minimises expected mean square error of the estimate each divide by different numbers – Henry Apr 28 '22 at 07:44

3 Answers3

15

Both are correct. As said by blooraven (+1), this is the same kind of correction as in the unbiased estimator for sample variance. The second formula is used with linear regression corrects for the number of degrees of freedom.

Notice that the second formula would not make sense in every context. Some models can be used with more features than samples, so the denominator would be zero or negative. In non-parametric models, or even some parametric ones (neural networks), it may be hard to say how many degrees there are and what exactly they are. Because of that in machine learning, to compare different models, you would almost always see the first formula with the denominator that is simply $n$.

Tim
  • 138,066
  • 9
    +1. To expand on the final sentence, the machine learning approach would be to calculate MSE on out-of-sample data, not in-sample data. – Dave Apr 27 '22 at 11:59
  • 2
    Dave's point is so important I would consider making it part of the answer. Otherwise it may seem that the unbiased version is correct and the "ML" version is an approximation while the "ML" version is correct, in the context in which it used, i.e. out of sample (e.g. cross-validation) error estimation. – Luca Citi Apr 28 '22 at 07:18
10

Assuming that the slide is talking about linear regression with one input variable, i.e. $$y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$$, the correct formula for MSE is: $$ \operatorname{MSE} = \frac{1}{n-2} \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2 \ . $$ To reiterate, for the specific case of a linear model with only one input variable the denominator must be $n-2$.

In the more general case when you have a linear model with $k$ input variables that is: $$ y_i = \beta_0 + \beta_1 x_{1i} + \beta_2 x_{2i} + \dots + \beta_k x_{ki} + \varepsilon_i \ , $$ then the MSE would be: $$ \operatorname{MSE} = \frac{1}{n-(k+1)} \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2 \ . $$

I am not aware of any model in which the denominator would be $n$. Usually, the denominator of $n$ is only possible when we know the population parameters $\beta_j$, in which case we are computing the true residual variance not estimating the residual variance.

blooraven
  • 429
  • can I trouble you to take a look at this page and this wikipedia page under the predictors section. In these cases, are they referring to something else when they talk about mean squared error? – tangolin Apr 27 '22 at 06:37
  • 4
    @tangolin I see what's going on! There's two senses in which the MSE is used: (i)as an estimator of the residual variance i.e. in the statistics sense ; (ii) as a way to evaluate your predictions, let's call it the machine learing sense. When you want to use the MSE as an estimator for residual variance in a statistical model, if you want an unbiased estimator you need to use the $n-k$ denominator. If you are merely using the MSE to as a loss function i.e. how much were my predictions wrong, you use n. – blooraven Apr 27 '22 at 14:39
  • @tangolin also see Dave 's answer – blooraven Apr 27 '22 at 14:40
  • The last sentence is incorrect "in which case we are computing the true residual variance not estimating the residual variance": you would still be estimating the residual variance because you are using a finite sample. The only difference is that you either assume that the model coefficients are known exactly or that they have been estimated on separate data. – Luca Citi Apr 28 '22 at 07:15
  • @LucaCiti I think you're right. Can you edit the answer? – blooraven Apr 28 '22 at 23:23
0

The population's mean square error (MSE) is only computable when the population's parameters are known. In this case, the formula for the MSE of {a population of size $N$} is: $$\displaystyle\sum^N_{i=1}\left(y_i-\hat{y}_i\right)^2*\frac{1}{N}$$

When we don't know the population's parameters, then we can only estimate the population's MSE. In this case, the {unbiased estimate of the MSE of a population} is estimated from sample size $n$ in {a model with $k$ variables}, according to this formula: $$\displaystyle\sum^n_{i=1}\left(y_i-\hat{y}_i\right)^2*\frac{1}{n-(k+1)}$$

moses.rivera100
  • 81