In a probability and statistics class, we are often told about the requirement of residuals being uncorrelated when fitting a regression model.
We are often told that the reason for this requirement is that the absence of this usually suggests low quality data.
But mathematically, is there a mathematical reason as to why residuals need to be uncorrelated? What happens if the residuals are in fact correlated?
Can someone please comment on this?
The requirement of residuals being uncorrelated is typically a heuristic, which is based on the following chain of assumptions:
If the regression model itself can be assumed to be approximately linear (actually affine due to the constant term), and the errors of deviation from this model are uncorrelated with zero mean and equal variance, then according to the Gauss-Markov theorem, the ordinary least squares (OLS) estimator will produce the vector of unbiased regression coefficients with lowest variance "envelope", within the class of linear unbiased estimators.
Since the model is assumed to be approximately linear (affine), then the residuals from using the estimated vector of regression coefficients will be linearly related to the vector of residuals, and these residuals should also enjoy small uncorrelated values.
Please note that the residuals computed from an OLS estimate in a linear (affine) model will not be identically distributed, due to the effect of Leverage.
Without the "nice" linearity and OLS assumptions mentioned above, i.e., for any general perhaps nonlinear model and/or nonlinear estimation procedure, the requirement of residuals being uncorrelated is typically a heuristic based on the notion of having extracted the most information out of the data. Uncorrelatedness is a heuristic proxy for this notion, not a rigorous mathematical one.
One exercise I give to students is to take a small data set where there is an insignificant relationship between Y and X. Then, simply duplicate the data set so that there are twice as many observations. You will notice that the coefficient estimates and R-squared value are identical to the original, but the p-value is smaller. Repeating this process, you can make the p-value as small as you want.
Now, this is obviously silly statistical practice, but what is specifically violated here? The answer is the uncorrelated errors assumption. All the other assumptions (linearity, homoscedasticity, normality), if valid for the process that produced the original small sample, remain valid for the process that produced the duplicated sample.
The point is that the big bad wolf indeed has teeth: gross violations of the uncorrelated errors assumption can have enormous effects on the validity of the inferences (intervals as well as tests).
Uncorrelated or independent residuals/observations/errors* are often assumed (but not required) for statistical inference like computing confidence intervals or p-values. If that assumption is wrong then the computed values are wrong (and methods that assume correlated observations would have been better).
See for instance the following time series that are randomly generated.
First 49 times two autocorrelated series with $cov(y_i,y_j) = 0.9^{|i-j|}$
Among these first 49 cases there are 20 cases with $p<0.05$ (shown in red text) and the p-values are underestimated (and significance is overestimated).
Second 49 correlated series without auto-correlation.
Here there is only one case with $p<0.05$ which is closer to the expected number.
It is not the residuals, but the error terms or observations that are assumed to be independent. The residuals will be correlated. See for instance the illustration in Why are the residuals in $\mathbb{R}^{n-p}$?
Strictly speaking, it is the error terms that are assumed to be uncorrelated in a standard regression model. The residuals are actually slightly correlated owing to their mutual dependence on the estimated parameters in the model. If the error terms in a regression analysis are correlated then it is possible to build this into the model, which is essentially what is done in a linear mixed model. Moreover, it is simple to test the assumption that the errors are uncorrelated using diagnostic tests; usually this would involve looking at correlation or auto-correlation estimates for the residuals.
If you don't account for correlated errors in a regression model (i.e., if they are actually correlated but you fit a model that assumes they are uncorrelated) then this will mean that your estimated variance for the coefficient estimators and residuals will sytematically under or over-estimate the true values. For example, if the error terms are positively correlated (which is the more common case) and you treat them as uncorrelated, you will tend to underestimate the true variance in the coefficient estimators in the model.
Correlated error terms are certainly not the big bad wolf of statistics --- they will not blow your house down and there are much nastier problems that can be encountered. The main thing that keeps them in check is the fact that diagnostic tests in regression can generally estimate residual correlation well, so they can be used to check the plausibility of underlying assumptions about the correlation of the error terms. If you find positive correlation in your error terms you can model this using a linear mixed model, where the additional random effects term in the model induces positive correlation in the overall error term and thereby generalises to a case that allows such correlation.
