8

I'm wondering what difference it makes whether we talk about bias-variance dilemma where fitting a regression line to the given dataset reduces bias and increases variance or whether we talk about overfitting where the model is fitted to the dataset and does not generalize to new datasets.

Essentially overfitting implies the same tradeoff between biasedness and variance doesn't it?

upabove
  • 3,137

2 Answers2

8

I think they're related, but not identical problems.

For example, its entirely possible to run into the bias-variance dilemma arguably well before a model is "overfit". For example, in survival analysis one might reasonably argue that a constant hazard (exponential distribution) is so unlikely that you should always "pay" for the protection of using a more flexible distribution like the Weibull distribution. There are a number of other examples where one might struggle with lessening the precision of an estimate well before you've really hit "overfit". Especially in fields where you're often working quite close to the null (environmental epidemiology comes to mind).

To me, overfitting also implies a problem with the interpretation of your model. If one has a regression equation like y = b1 + b2*x + b3*(x^2) + b4*(x^3) + b5*(x^4) + b6*(x^5) you have a problem beyond the bias/precision tradeoff, and into how one meaningfully interprets higher order polynomial relationships into the field.

Fomite
  • 23,134
  • Hmm, to me, the situation you describe (exponential distribution being too simple enough to adequately model the situation) is an example of underfitting. If that's true, then an answer could be given essentially agreeing with the premise, under the condition that "underfitting" be added. i.e. overfitting and underfitting implies the same tradeoff as bias-variance. – ijoseph Aug 16 '18 at 19:50
  • Although I also agree that overfitting is related to interpretability issues. – ijoseph Aug 16 '18 at 19:51
5

I view "overfitting" as an example of a bias-variance trade-off (I wouldn't call it a "dilemma") that has gone too far toward the variance end.

The usual example I have in mind for the bias-variance trade-off is the case of a linear model in which all covariates have some effect, but where it would best to drop those whose effects are negligible, incurring some bias in order to reduce the variance.

The usual example I have in mind for "overfitting" is to imagine that the truth is contained within the class of models being considered, so that things could be unbiased, but then the class of models is expanded through added parameters, making it overly flexible, so that the fitted model resembles the observed data quite closely but not necessarily the underlying population or process.

I actually quite dislike the term "overfitting". I'd rather say "fitting too complex a model".

Karl
  • 6,197
  • 1
    I'm not sure that a model being "exceedingly flexible" and having little bias are necessarily related. – cardinal Oct 16 '11 at 00:27
  • 1
    @cardinal - agreed. I was thinking: if the truth were contained within the class of models being considered, things could, in principle, be unbiased, but if the class were made considerably more flexible through added parameters, that would lead to what is called "overfitting". In the bias-variance tradeoff, the bias part is often a result of a simplified model that only approximates the truth (eg all covariates in a regression have some effect, but we drop those whose effect is negligible. – Karl Oct 16 '11 at 00:33
  • Yes, I agree with your last comment, and it might deserve to be part of the main text of the answer. That said, it is the leading phrase in your comment that causes most of the indigestion. :) – cardinal Oct 16 '11 at 00:40
  • 1
    @cardinal - Yes, thanks. I appreciate your comments and have revised my answer so that it hardly resembles my original answer. – Karl Oct 16 '11 at 01:34