Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [overfitting]

Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.

Models that involve complex polynomial functions or too many independent variables may fit particular samples' covariance structures overly well, such that some existing (and any potential, additional) terms increase model fit by modeling sampling error, not systematic covariance that is likely to replicate or represent theoretically useful relationships. When used to predict other data (e.g., future outcomes, out-of-sample data), overfitting increases prediction error.

The Wikipedia page offers illustrations, lists of potential solutions, and special treatment of the topic as it relates to machine learning. See also:

Leinweber, D. J. (2007). Stupid data miner tricks: Overfitting the S&P 500. The Journal of Investing, 16(1), 15–22. Available online, URL: http://www.finanzaonline.com/forum/attachments/econometria-e-modelli-di-trading-operativo/903701d1213616349-variazione-della-vix-e-rendimento-dello-s-p500-dataminejune_2000.pdf. Accessed January 6, 2014.

Tetko, I. V., Livingstone, D. J., & Luik, A. I. (1995). Neural network studies. 1. Comparison of overfitting and overtraining. J. Chem. Inf. Comput. Sci. 35(5), 826–833. doi:10.1021/ci00027a006.

983 questions
115
votes
21 answers

What's a real-world example of "overfitting"?

I kind of understand what "overfitting" means, but I need help as to how to come up with a real-world example that applies to overfitting.
user3851283
  • 307
33
votes
2 answers

Can overfitting and underfitting occur simultaneously?

I am trying to understand overfitting and underfitting better. Consider a data generating process (DGP) $$ Y=f(X)+\varepsilon $$ where $f(\cdot)$ is a deterministic function, $X$ are some regressors and $\varepsilon$ is a random error term…
Richard Hardy
  • 67,272
10
votes
1 answer

Is a saturated model a special case of a overfitted model?

I am trying to make sense of what a saturated model is. AFAIK it's when you have as many features as observations. Can we say a saturated model is a special-case of an extremely overfitted model?
Ricardo Cruz
  • 1,581
8
votes
1 answer

Overfitting on the loss graph, but not the accuracy graph

I am looking at learning curves (CNN for text classification, which is based on this paper) and trying to play with regularization to prevent overfitting. This model uses L2 regularization and dropout. What is interesting is that by looking at the…
Yuri
  • 545
  • 1
  • 5
  • 11
7
votes
2 answers

How does one most easily overfit?

This is a weird question, I know. I'm just a noob and trying to learn about different classifier options and how they work. So I'm asking the question: Given a dataset of n1-dimensions and n2-observations where each observation can be classified…
MetaStack
  • 473
5
votes
1 answer

Cannot overfit on the IRIS dataset

I am playing with the IRIS dataset and want to see underfitting and overfitting in action. I am using a multilayer perceptron (2 layers). It is pretty easy to underfit (see the plot below), but I am having problems with overfitting. The dataset…
Yuri
  • 545
  • 1
  • 5
  • 11
3
votes
1 answer

Misconception about overfitting

I hear countless times at my job that the gap between train and test implies that there's overfitting. Some people go as far as saying that the goal of model selection is to reduce the gap between train and test. The same people also say that a…
Nick Corona
  • 127
  • 6
2
votes
1 answer

Why validation accuracy starts to increase after overfitting?

I'm training a model on a small dataset of images. following are the curves of accuracy, f1 score and auc score. it's clear that the model is overfitting, however I don't understand why after sometime its validation results starts to improve. Can…
Ines
  • 31
2
votes
1 answer

Why overfitting might not happen?

I am working with a pretty big dataset (800k samples) and I solve a classification problem. What puzzles me is that models (CNN and MLP) with ~3000 and 3000000 parameters have pretty much the same (and decent) performance. The bigger model does not…
Yuri
  • 545
  • 1
  • 5
  • 11
1
vote
3 answers

What is overfitting while building model?

What exactly is overfitting while building models ?
SR1
  • 31
1
vote
1 answer

What's is the training data here?

Can somebody explain me why this classifier is giving a loss equal to cero? I don't get the example SourceText
Stephen
  • 257
1
vote
0 answers

Is there life after over-fitting?

At this point in a talk by Nando de Freitas, there is an answer to an audience question, about how theory has got left behind in statistics, but theory is still important, and he gives an example where there was an observed phenomenon in machine…
Darren Cook
  • 2,107
  • 1
  • 14
  • 26
1
vote
0 answers

Overfitting a mixed effect model

I wonder if the following mixed model suffers of overfitting Note: The following is just an hipotethical model to examplify the model construction. Research question: what affect apple production? Experiment design: I have 100 indipendent sites…
Dave
  • 11
  • 1
1
vote
1 answer

Looking for automatic heuristics/techniques that can produce a measure for overfitting

Are there heuristics/techniques that can be employed to guard against overfitting of a model that do not require/employ human inspection of the learned model's performance. So for example, they would look at the performance of a predictor and come…
user1172468
  • 2,035
1
vote
1 answer

Is it valid to reduce noise in the test data from noisy experiments by averaging over multiple runs?

I have data from a biological (fMRI) experiment that was previously evaluated with a different model in a machine learning fashion, using a training and a validation set in a cross-validation routine for finding regularization parameters and a test…
kapibarasama
  • 155
1
2