11

Recall the model for simple linear regression $$ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i. $$

I am reading up on the standard error of the coefficients $\beta_0$ and $\beta_1$. As an experiment, I generated some linear data using $\beta_0 = 1$, $\beta_1 = 2$, and added some Gaussian noise with unit variance. So then, when I fit the data using the lm function and used the summary function to examine the model, I have the following output: \begin{align} \hat \beta_0 & = 1.21054 \quad \text{with Std. Error} = 0.11508, \\ \hat \beta_1 & = 1.87723 \quad \text{with Std. Error} = 0.09844. \end{align}

So how do I interpret the standard error values? For instance, take $\hat \beta_0$; precisely what is $0.11508$ telling me?

Obviously, if I ran the simulation a second time, this time adding Gaussian noise with a higher amount of variance, the standard error would increase as the extra variance in the noise shows up as an increase in the standard error of the coefficients. But, if we consider the first simulation in isolation, then what does this value of $0.11508$ mean?

Valerio
  • 37
ManUtdBloke
  • 893

4 Answers4

16

The standard error is the square root of an estimate of the sampling variability of $\hat\beta_j$ as an estimator of $\beta_j$, or $\sqrt{\widehat{Var}(\hat\beta_j)}$.

As this is many things in one sentence, step-by-step:

  1. "Square-root": should be self-explanatory, to turn a variance into a standard deviation (that turns out to be what we need in, for example, t-statistics and confidence intervals).
  2. "$\hat\beta_j$ as an estimator of $\beta_j$": we use the LS estimator to estimate the unknown parameter $\beta_j$.
  3. To do so, we make use of a sample from the underlying population. Had we drawn another sample (or were to draw a fresh one tomorrow, etc.) we would get another estimate $\hat\beta_j$. This is the source of sampling variability. We may summarize that variability through the variance, $Var(\hat\beta_j)$. An expression for this variance may be found, e.g., here.
  4. "An estimate of the sampling variability": $Var(\hat\beta_j)$ depends on unknown quantities (like the variance of the Gaussian noise that you generated), which must therefore be estimated, as captured by the formula $\widehat{Var}(\hat\beta_j)$. A formula for this estimator is, for example, given here, or, more introductory, here.
Christoph Hanck
  • 33,180
  • 1
    My question is about what this particular value of $0.11508$ means. I know we will get different outcomes if we draw other samples. But for a single sample in isolation: what does it tell me when the coefficient is $1.21054$ with a standard error of $0.11508$? – ManUtdBloke Sep 02 '20 at 12:32
  • 4
    In and of itself, it does not tell us too much, but it becomes interesting when coupled with the distribution of the LS estimator, which is normal under suitable assumptions. Since 95% of the realizations of a normal random variables are (roughly) fall into two standard deviations around the mean and since LS is unbiased (again, under suitable assumptions), we may, e.g., use the point estimate and the standard error to obtain a confidence interval, which is heuristically often interpreted as a range of plausible estimates for the true parameter. – Christoph Hanck Sep 02 '20 at 12:35
  • 1
    E.g., https://stats.stackexchange.com/questions/351320/can-we-reject-a-null-hypothesis-with-confidence-intervals-produced-via-sampling/351333#351333 https://stats.stackexchange.com/questions/253915/what-is-the-intuition-behind-using-pivotal-quantities-to-find-confidence-interva/253947#253947 https://stats.stackexchange.com/questions/183813/why-do-statisticians-prove-asymptotic-normality/183828#183828 – Christoph Hanck Sep 02 '20 at 12:36
  • 3
    Alternatively, point estimate/standard error is the t-statistic for the null hypothesis that the true coefficient is zero. So with your numbers, the t-statistic would far exceed two, the critical value at the 5% level, so that you would reject the null at that level. – Christoph Hanck Sep 02 '20 at 12:39
  • The answer of Emil Bode below indicates that we can give confidence intervals based on these standard errors. Is this user incorrect? – ManUtdBloke Sep 06 '20 at 18:05
  • 1
    No, (e.g.) the first link in my comment above recalls how to build a confidence interval. What I want to alert to is that the language used there might indicate that confidence intervals are interpreted that is not afforded by their definition. – Christoph Hanck Sep 07 '20 at 08:05
5

It’s the usual definition of the standard error: the (estimated) standard deviation of the sampling distribution of $\hat{\beta}_0$.

If you were to replicate the work many times with new observations, you would get a distribution of values. Sometimes it would be higher than you observed this time, sometimes lower.

We use the standard error in parameter inference. Being loose, if the p-value on the parameter is less than $0.05$, corresponding to a point estimate about $2$ standard errors above or below $0$, then we might say that the population parameter is not zero, so that variable has a measurable impact on the outcome.

(There are all sorts of caveats about p-values, and discussing them really warrants a separate question (or a master’s degree in statistics).)

Dave
  • 62,186
  • My question is about what this particular value of $0.11508$ means. I know we will get different outcomes if we draw other samples. But for a single sample in isolation: what does it tell me when the coefficient is $1.21054$ with a standard error of $0.11508$? – ManUtdBloke Sep 02 '20 at 12:32
  • 1
    It has to do with hypothesis testing and creating confidence intervals, same as usual. – Dave Sep 02 '20 at 13:24
4

If

\begin{align} \hat \beta_0 & = 1.21054 \quad \text{with Std. Error} = 0.11508, \\ \hat \beta_1 & = 1.87723 \quad \text{with Std. Error} = 0.09844. \end{align}

it means that the range of values for the coefficient estimates are

$$\hat \beta_0 = 1.21054 \pm 0.11508$$

and

$$\hat \beta_1 = 1.87723 \pm 0.09844$$

In other words, you can be confident that $\beta_0$ can take on values between $1.09546$ and $1.32562$.

As for your repeated question of

But for a single sample in isolation: what does it tell me when the coefficient is 1.21054 with a standard error of 0.11508

it is not relevant because the $\beta$s (and therefore the $\sigma(\beta)$s) are computed based on the entire sample set, not from one specific observation. A $\beta$ relates the entire $y$ sample set with the entire sample set being input for (one of) the corresponding $x$ vector.

I think what you mean to ask is, what does $\hat \beta_0 = 1.21054 \pm 0.11508$ mean for the output $\hat{y}$ of my fitted model if a new sample $x_i=0.2$ is observed. Well, since $y = \beta_0 x + \beta_0 x + \epsilon$, then the predicted output given that new input is

$$\hat{y}_i = (1.21054 \pm 0.11508) \times 0.2 + (\beta_1 \pm \sigma(\beta_1))\times 0.2 + \epsilon$$

develarist
  • 3,917
  • 1
  • 21
  • 52
  • I don't understand $$\hat \beta_0 = 1.21054 \pm 0.11508$$ $\hat \beta_0 = 1.21054$... I think I know what you're trying to say, but the notation here doesn't really make sense. – Do not reinstate Monica Sep 02 '20 at 14:48
  • In other words, you can be confident that $\beta_0$ can take on values between $1.09546$ and $1.32562$.--- What I am really interested in is precisely how confident can I be that $\beta_0$ will be within that range? – ManUtdBloke Sep 02 '20 at 15:45
  • Thsi answer and your last question seem to have a somewhat Bayesian flavor. The standard error cannot be used to make such confidence statements. It would be too broad to explain the Bayesian idea here, though, but there are many excellent posts on this site on this approach. – Christoph Hanck Sep 03 '20 at 04:26
  • @ChristophHanck The answer of Emil Bode below indicates that we can give confidence intervals based on these standard errors. Is this user incorrect? – ManUtdBloke Sep 06 '20 at 18:04
3

What you created here is a model, that tries to reflect reality. But of course, unless we are exceptionally lucky, the model will never reflect reality perfectly. And the standard deviations reflect how confident the model is about itself.

Your question stated that you generated data with $\beta_0 = 1$ and $\beta_1=2$. Those numbers are the reality your model tries to reflect. Now suppose you didn't tell us those values, just your model. What can we say about your input?

The model tells us the most likely values are $\beta_0 = 1.21042$ and $\beta_1=1.87223$. But could it be that you that the actual values you put in (the reality) were $1.2$ and $1.9$? Well, therefore, we have to look at the standard deviation.

With the given standard deviations, the model tells you it's $68 \%$ sure the true value of $\beta_0$ is in the range $1.09546 - 1.32562$ (minus 1 sd and plus 1 sd). And it's $95 \%$ sure the true value is in the range $0.98038 - 1.4407$ (2 sd away). For $\beta_1$, we can do a similar calculation. That means the numbers $1.2$ and $1.9$ are very reasonable guesses, but that $1$ and $2$ are also not too outlandish.

Now, in reality, we often don't have access to the true values of $\beta_0$ and $\beta_1$. We can just take measurements and make the best model we have. Or sometimes, theorists will develop a theory that has to be tested in reality to check whether the model is right or wrong.

As an experimental physicist, you'll run some experiments and maybe get the same values you got. You'll make a model and can publish this to show that a theory that predicts $\beta_0 = 0$ and $\beta_1=5$ is most definitely wrong (if you can prove your experimental setup is correct). The values you got of $1.21$ and $1.87$ are basically your best guesses as to what the true values could be. But a theory that predicts $\beta_0=1$ and $\beta_1=2$ may well be correct.

Until you come up with a more sensitive experiment. Suppose you do the same and get a model that shows: \begin{align} \hat \beta_0 & = 1.19554 \quad \text{with Std. Error} = 0.01279, \\ \hat \beta_1 & = 1.88341 \quad \text{with Std. Error} = 0.02369. \end{align}

These values align quite well with your earlier result (showing there was likely no systemic error in your first experiment). But they have much narrower standard deviations, and now also show the theory with $\beta_0=1$ and $\beta_1=2$ is also wrong. But the guesses of $1.2$ and $1.9$ are still holding.

Valerio
  • 37
Emil Bode
  • 131