3

I understand that the standard error of the regression estimate is used over the variance because it's in the same units as the predictor, but I'm wondering if there may be a more intuitive statistic.

When I'm doing regression analysis, I don't really have a good sense of what an acceptable standard error is, other than that it should be low relative to the average value of $Y$. If instead of

$$s_{est}=\sqrt{\sum{(Y-Y')^2}\over {N-p}},$$

we used:

$$\sum\sqrt{(Y-Y')^2}\over {N-p}$$

then the interpretation would simply be the average absolute difference between $Y$ and $Y'$. Is there an easy way to interpret the standard error? If not, why don't we use the average absolute deviation?

user123965
  • 699
  • Why is the average absolute difference more intuitive than the average squared difference? In practice we use variances etc because it makes the number crunching easier. – Richard Redding Apr 21 '17 at 17:23
  • @RichardRedding - more intuitive than the square root of the average squared distance. What I'm trying to suggest is that dividing by $N-p$ before square rooting makes it harder to interpret - at least as far as I can see. – user123965 Apr 21 '17 at 18:30
  • @GeoMatt22 Yes, or have the summation outside the root. I'll edit it. – user123965 Apr 21 '17 at 18:32
  • 1
    Standard error is not an error in the sense of your operating domain. It is a statistical measure under Gaussianity. You may use another error measure after the estimation process. There is nothing wrong with that. – Cagdas Ozgenc Apr 21 '17 at 18:46
  • A good way to find an "intuitive" statistic is to study the ones that are commonly used, with such care and attention that you develop an intuition for what they mean and what they can accomplish. That has the salubrious effect of broadening one's experience rather than narrowing one's options. – whuber Apr 21 '17 at 19:05

1 Answers1

2

In least squares the sum square error is minimized by the regression coefficients $$\sum{(Y-\hat{Y})^2}$$ If $N$ and $p$ are considered fixed (i.e. the model and data are fixed), this is the same as minimizing the root mean square error.

If instead you did LAD regression, then the regression coefficients would minimize $$\sum{|Y-\hat{Y}|}$$ and in this case it would make more sense to report the mean absolute deviation.

As least squares regression is the "default", typically you will see the root mean square used as the indicator of "goodness of fit".

More theoretically, least squares gives a MLE for the coefficients (and error-variance) under the assumption of Gaussian errors.

For Laplace errors, which are fatter-tailed, least absolute deviations regression gives a MLE for the coefficients (and error mean-absolute-deviation). Because it allows for fatter-tailed errors, LAD can be used to provide robustness to outliers.

GeoMatt22
  • 12,950
  • Does the RMSE have an intuitive interpretation, or is there a similar statistic that does? The reason I like the MAD is that it has a very intuitive explanation, e.g. predictions are off by 1000 on average, whereas the RMSE needs a little extra work to interpret meaningfully – user123965 Apr 21 '17 at 18:44
  • It is the stdev. of the residuals PDF (e.g. assuming a normal distribution). So it is no more or less intuitive than any stdev. MSE (i.e. variance) is more fundamental. For a physical analogy, perhaps thinking in terms of moment of inertia will help? (radius of gyration, rather) – GeoMatt22 Apr 21 '17 at 19:12