Modeling error (especially sampling error) instead of replicable and informative relationships among variables improves model fit statistics, but reduces parsimony, and worsens explanatory and predictive validity.
Models that involve complex polynomial functions or too many independent variables may fit particular samples' covariance structures overly well, such that some existing (and any potential, additional) terms increase model fit by modeling sampling error, not systematic covariance that is likely to replicate or represent theoretically useful relationships. When used to predict other data (e.g., future outcomes, out-of-sample data), overfitting increases prediction error.
The Wikipedia page offers illustrations, lists of potential solutions, and special treatment of the topic as it relates to machine learning. See also:
Leinweber, D. J. (2007). Stupid data miner tricks: Overfitting the S&P 500. The Journal of Investing, 16(1), 15–22. Available online, URL: http://www.finanzaonline.com/forum/attachments/econometria-e-modelli-di-trading-operativo/903701d1213616349-variazione-della-vix-e-rendimento-dello-s-p500-dataminejune_2000.pdf. Accessed January 6, 2014.
Tetko, I. V., Livingstone, D. J., & Luik, A. I. (1995). Neural network studies. 1. Comparison of overfitting and overtraining. J. Chem. Inf. Comput. Sci. 35(5), 826–833. doi:10.1021/ci00027a006.
