7

I am working on a project in which I am using several independent variables to "predict" the values of an outcome using linear regression.

In R this is done quite simply as

model  <- lm(outcome ~ predictor1 + predictor2 + predictor3)
fitted <- model$fitted.values

I am interested in the difference between the predicted values and the actual values - i.e. how accurate the predictors are. 

residuals <- model$residuals

My question relates to the relationship between residuals and outcome.

Samples with lower values of outcome tend to have negative values for residuals, and vice versa for samples with high outcome values.

Plotting the values against one another is the simplest way to see this:

Various plots

The $R^2$ for the original LM (outcome ~ predictors) is 0.42, the $R^2$ between residuals and outcome is 0.58, and the $R^2$ between fitted and outcome is 0.39.

What could explain the phenomenon? Why would samples with high outcome tend to be predicted lower than they actually are, and vice versa for lower values of outcome? Or indeed, am I missing something conceptually here?

Many thanks for your input

Edited (13.08.20) to include an updated plots and terminology (now use "residuals" rather than "difference") - but in essence the questions remains the same. Thanks all for the input so far.

Luke
  • 205
  • If I'm understanding you correctly, your line shouldn't be a worse fit at the ends (at least not as much as this). – Justin Bozonier Aug 20 '13 at 11:11
  • 1
    Could you add a plot of your regression against the original data? (by the way, you can also directly use model$residuals or resid(model) to find the residuals) – nico Aug 20 '13 at 11:25
  • This post discusses the question, if it makes sense to study the residuals vs. the original variable (outcome). – COOLSerdash Aug 20 '13 at 14:44

4 Answers4

6
  • Usual conventions

The usual conventional name and definition are

residuals = outcome $-$ fitted

Similarly, the usual conventional plot is residuals (y axis) versus fitted (x axis).

In R, given something like 

mymodel = lm(outcome ~ predictor1 + predictor2 + predictor3)

then 

plot(mymodel)

gives that plot as one of a portfolio. That's usually a much easier plot to think about your plot. You can also plot outcome versus fitted. The first is critical, in exposing weaknesses of the model, and the second is positive, in focusing on the strength of the model.

  • What you originally did

The usual set-up is that of observed $y$, fitted $\hat y$, and residual $e$ linked by

$y = \hat y + e$

With this set-up a plot of $y$ versus $e$ has an overall slope of $+1$. There is variability around that overall slope, but it is not correlated on the whole with the residuals. Your original difference variable contained negated residuals, so the overall slope became $-1$.

  • Note on $R^2$

In your case, note that the two values of $R^2$ add to 1, i.e. $0.42 + 0.58 = 1$, which follows from the fact that the proportion of variance "explained" by the model and the proportion of variance "not explained" are mutually exclusive. (The correlation between residual and fitted is zero, so the covariance term is zero.)

  • Summary

The spirit of your original plot (now deleted) was right, but it's better just to plot residuals versus fitted. Indeed what you did puzzled or confused some people because they misread a procedure that is not standard for one that is. The pattern of your plot makes sense and is not incorrect or anomalous.

Nick Cox
  • 56,404
  • 8
  • 127
  • 185
  • Thanks for the clarification on the usual procedure and terminology here. I have updated my question accordingly, but think that Peter has hit the nail on the head in terms of clarifying the trend I was confused about. – Luke Aug 20 '13 at 15:46
  • 2
    Indeed. I followed him and was just filling in details, notably why the pattern is linear on average. – Nick Cox Aug 20 '13 at 15:54
5

Basically, it's because the regression isn't perfect.

Suppose you had purely random data - no relation between the dependent and independent variables. Then the best prediction of the DV for every subject would be the mean of the DV.

Suppose you had a perfect relationship; then you be able to exactly predict the DV.

In reality, it's always somewhere in between, and the predicted values are between the mean and the actual values.

Peter Flom
  • 119,535
  • 36
  • 175
  • 383
3

This figure shows that there seems to be a significant variable you are missing.

Because the residuals have a clear trend, i.e containing important information.

If you have not more variable on the data, maybe you could try the interactions.

Vincent
  • 2,486
  • It is quite possible that there is structure in the residuals. But I think you are misinterpreting this plot as, or as if it were, a plot of residuals $e$ versus fitted $\hat y$, with axes reversed and residuals negated. But it's a plot of residuals and observed $y$. An $R^2$ of 0.58 could not be observed for a residual plot, for which $R^2$ is necessarily 0. – Nick Cox Aug 20 '13 at 12:01
  • 1
    This answer refers to a previous plot, now deleted. – Nick Cox Aug 20 '13 at 19:37
3

The concept is not new and is called Regression to the Mean, or Regression towards the Mean, see here for history and detail. In fact the story goes that this is how regression analysis (linear models, least squares, etc.) ended up being called "regression".

Greg Snow
  • 51,722