1

I have a data set where some extreme, but not nonetheless important observations are present which prompts violation of the linear regression assumptions of normality and constant variance. The assumption of normality I read here: Regression when the OLS residuals are not normally distributed, is of less concern as long as we're not concerned with p-values or intervals.

The violation of constant variance however is discussed here: What does having "constant variance" in a linear regression model mean?, as well as here: Regression when the OLS residuals are not normally distributed, where Michael R.Chernick in the comments of his response writes

log transformation or Box-Cox with small lambda shrink the tails. That can work for some heavytailed and skewed distribution. I don't know what if any transformations will work for very heavy-tailed distributions.

As such I tried Box-Cox transformation, with optimal lambda = 0.1414141 and also using first difference rewriting of the response variable ($y_t - y_{t-1}$). I plotted the results of the optimal Box-Cox transformation:

BoxCox fitted model

This does not provide a solution to my situation, and the assumption of constant variance remains violated.

I would like some pointers towards what I can, if any, to fix this alternatively how I can model my data using other regression techniques.

Edit My modelling is about quantifying uncertainty of an agnostic prediction model, in a highly stochastic environment (travel times in traffic). For the two data points in the graph that normally would be considered outliers, the agnostic prediction model made a reasonable prediction. I have checked this by comparing the predictions versus the actual outcome. That motivates me to not treat them as outliers, as I assume the data fed forward to the prediction model was viable.

The agnostic prediction model have access to GPS position and speed when making its predictions about arrival times. I don't have access to the same data, but rather what I'm using here is the previous travel time as a modeling feature. The reason behind those data points being so extreme is unknown to me.

OLGJ
  • 317
  • With two extreme outliers among what seem to be over 1000 observations, I'm not sure that a single simple linear regression will adequately represent your data. You say "some extreme, but not nonetheless important observations are present." It might help to describe why those observations should be included in your analysis. Why are those values so extreme? Please provide that information by editing the question, as comments are easy to overlook and can be deleted. – EdM Mar 23 '23 at 15:57
  • Those two outliers are so far away from everything else after your transformation. Why do you think they should be modeled by the same data generating process? – wzbillings Mar 23 '23 at 16:11
  • I tried to provide an explanation about why I think they shouldn't be treated as outliers @EdM, please let me know if helps to clarify the situation. – OLGJ Mar 24 '23 at 09:03
  • @wzbillings I have made an edit to the post where I provide an explanation. – OLGJ Mar 24 '23 at 09:03

1 Answers1

1

From the plots the problems seem to boil down to two outliers (OK I get that these plots are from after Box-Cox transformation and before things may have looked worse), potentially leverage points. This may have a rather detrimental effect on the regression estimation, but can be handled well using robust regression. A state of the art method for this is the MM-estimator, as performed by R-function lmrob in package robustbase. Note that this will estimate a hopefully good regression for the non-outliers; the outliers, if important and correct, should be separately interpreted.

Proviso: Obviously I don't see diagnostic plots from before transformation, but the outliers may affect the Box-Cox as well. Using robust regression with the same Box-Cox lambda is probably fine, but also another or no transformation may (or may not) work well with robust regression.

Christian Hennig
  • 23,655
  • If MM-estimator estimates a "hopefully good regression for the non-outliers", does this mean the outliers are essentially disregarded by the model? If so, is there a reason to not just exclude them? – OLGJ Mar 24 '23 at 09:07
  • @OLGJ The MM-estimator will weight outliers down in a continuous manner, i.e., some observations will be given nonzero but low weight, some almost full weight, some zero weight. Particularly, it will determine these weights in a data-based manner. The outliers still have effect insofar that they are taken into account for the workings. The thing is that outlier identification based on OLS-residuals is unreliable as there may be masking effects and leverage points may not show large residuals but may have large effect. So you shouldn't think you know exactly what the outliers are based on OLS. – Christian Hennig Mar 24 '23 at 10:41
  • A major reason for running robust regression is that its diagnostic plots are more informative/reliable than those for OLS. – Christian Hennig Mar 24 '23 at 10:44
  • @OLGJ Furthermore, the distributional theory behind the tests run in MM-regression is robust against outliers, too, whereas OLS theory is invalidated by removing outliers in a data-based manner. – Christian Hennig Mar 24 '23 at 14:13