5

Maybe a duplicate-- but I am not asking about least squares vs minimizing absolute deviations. Just simply, what if I chose to minimize the sum of residuals:

Min $\sum_i^n (Y_i-\beta_0 - \beta_1X_i)$ ?

If I try to minimize this, the second order conditions are all zero. Is there a solution to this? Would the strategy be to pick $\beta$'s so that they make the sum of residuals negative infinity?

Dave
  • 62,186
Steve
  • 651
  • 5
    The sum of residuals has no interpretation as a loss function and hence is inappropriate as a criterion to be minimized. The only loss functions that make sense are those that increase in value as the fitted values get further from the observed values. – Gordon Smyth Nov 25 '22 at 22:49
  • This post could be useful, https://stats.stackexchange.com/a/274745/28746 as regards the desirable properties of a loss function. – Alecos Papadopoulos Nov 26 '22 at 20:26

2 Answers2

11

You start to reward the model for missing high, and the higher the miss, the better the loss. Indeed, you are correct to point out that the loss can be driven toward $-\infty$ as you make the intercept larger and larger.

Indeed, there is no minimum to the loss function, so slope and intercept parameters giving the minimum value do not exist.

Dave
  • 62,186
1

$\sum_i^n (Y_i-\beta_0 - \beta_1X_i) = 0$ to equivalent a regression line where $\bar Y=\beta_0 + \beta_1 \bar X$ i.e. any (non-vertical) line which passes through the mean point.

One such line is the ordinary least squares result. But all the others do so too.

Another is $\beta_1=0, \beta_0=\bar Y$, i.e. estimating a constant value for $\hat Y_i$ that ignores the $X_i$, but you can usually get a better fit than that.

You could aim to minimise the sum of the absolute deviations subject to passing through the mean point if you wished, but it would be analytically more complicated.

Henry
  • 39,459
  • 3
    What about values below zero? – Dave Nov 26 '22 at 02:38
  • 3
    Crazy though it might seem, the OP is minimizing the sum of residuals rather than setting the sum to zero. – Gordon Smyth Nov 26 '22 at 03:40
  • 6
    That is crazy. Perhaps I should delete my answer and replace it with one saying "Every time you increase $\beta_0$ by $1$ you decrease that sum by $n$, so you can easily make the sum of residuals as negative as you want" – Henry Nov 26 '22 at 05:49
  • There seems to be an answer saying just about that. – Dave Nov 27 '22 at 05:44
  • Just want to make it clear, I do understand that this is 'crazy', I was just thinking of it as a pure thought experiment (or showing precisely why it is crazy) – Steve Nov 27 '22 at 23:09