0

Cross-validation involves splitting data in training and test sets (several times) and assessing the goodness of applying the fitted coefficients from the training set to the test set. I was wondering which goodness measures can be used there.

Using regular likelihood, r-squared, adjusted r-squared etc is discouraged for robust regressions because of being subject to potentially being dominated by outliers. Even weighted r-squared should only be used as one of several other measures, according to this answer. Unfortunately the answer does not elaborate which others are available.

Related questions

This question asks about a closely connected issue - assessing the goodness of a fit in a test set corresponding to a robust regression model. Unfortunately, the accepted answer only addresses how a fitted R model can be applied to a test set and remains silent about what goodness measures to use.

This question also asks about goodness for robust regressions, but in a different context. An answer points out that the comparison the question asks about as stated cannot be done because of different data sets being used. A comment actually suggests cross-validation, but does not specify how.

Some details:

I am doing the robust regression with statsmodels.robust.robust_linear_model() from Python (version 3.10.7, statsmodels version 0.13.2) with Huber's norm (the default in the package; using any of the other options does not make a huge difference in my case.).

0range
  • 131
  • 5
  • What do you care about, and what about it is not satisfied by classical approaches? Let’s start there. // There’s nothing wrong with estimating a model one way and evaluating it another way. – Dave Oct 18 '22 at 16:01
  • @Dave Thanks. I would like to follow the best practice for robust regressions. The classical approach suggested in the link (RMSE) will, afaik, not work well with outliers, which is why I use a robust regression. (The example in the link seems to only have multicolinearity to show that RMSE will not always prefer models trained with OLS). Another answer in the same link suggests Mean Absolute Errors, as does this DS:SE answer, but I am unsure if there are other options or if this is the recommended best practice. – 0range Oct 18 '22 at 16:22
  • But what do you value in your predictions? – Dave Oct 18 '22 at 16:30
  • @Dave I am not 100% sure I understand what you mean by that. If you're asking about why I do a regression at all: Because I want to know if certain terms are (with a certain confidence) predictors and what their effect is (how large, what direction). If you're asking about why I want to do cross-validation: because I'm worried about overfitting, which could interfere with both size/significance and direction of coefficients. But I have the feeling that this is not quite what you're asking. – 0range Oct 18 '22 at 16:52
  • 1
    (Disclaimer: I never looked for any literature about this.) One possibility would be to look at robust analogues of the goodness-of-fit measures, for example use a robust variance (iqr, mad, ...) analogue to RMSE, or trim a fraction of worst prediction errors. Those would be measures how well the model fits "core" observations ("inliers"). – Josef Oct 19 '22 at 17:17
  • 1
    That would still leave it open, whether a better model would also fit "outlying" observations, i.e. extreme observations are not outliers, instead the model is misspecified. – Josef Oct 19 '22 at 17:20
  • 1
    Aside, I have used MAD instead of variance/RMSE to evaluate Monte Carlo simulations, if there are (optimization, numerical) problems with a few random samples. – Josef Oct 19 '22 at 17:23
  • 1
    https://www.statsmodels.org/dev/tools.html#measure-for-fit-performance-eval-measures – Josef Oct 19 '22 at 17:26
  • @Josef Thanks, this was very helpful. If you repost this as an answer, I will accept it. (Sorry for not mentioning this earlier.) – 0range Nov 21 '22 at 01:16

0 Answers0