6

Parsimony is often defined as the minimisation of unnecessary parameters or explanatory variables in a model. But models also have structure - functional forms that can change. Between two models that have the same number of parameters, is it possible to state that one model is more parsimonious than another? Another way of saying this might be: is there a way to quantify the simplicity of an elementary function?

For example, $f(x)=1-cos(x)$ and $f(x)=\frac{x^2}{2}$ are a very similar function between about -0.5 and 0.5:

1-cos(x) and x^2

They could be equally good models of a process acting over that limited domain. Is there any way of stating that one is simpler than the other?

naught101
  • 5,453
  • 1
    I don't know if there is a consistent, sensible way to quantify "complexity" in this sense. That said, I think there are a number of heuristic methods one could use to form decision rules to make these decisions within the context of a particular domain. For example, in some contexts an investigator may decide that a model with P main effects variables is more parsimonious than another model that also has P variables, but of which say K are interaction terms. That's a pretty contrived example, I admit, but I think the general point stands – Ryan Simmons Feb 06 '15 at 04:18
  • To refer back to the two functions you use as an example, an investigator may decide that the periodicity of the cosine function represents added complexity (or, conversely, that the asymptotic behavior of x^2/2 represents added complexity). There are ways to quantify this, but this wouldn't represent a broader theorem, but rather a simple context-based heuristic. – Ryan Simmons Feb 06 '15 at 04:21
  • I was going to suggest minimum description length, but if I understand you correctly, you're looking for a measure of complexity of a function regardless of the observed data? – Anthony Feb 09 '15 at 19:37
  • @Anthony: yes, independent of the data (but assuming that the data set is the same for each model) – naught101 Feb 09 '15 at 22:38
  • @Anthony: but MDL is really interesting, thanks for the pointer. – naught101 Feb 09 '15 at 23:28

2 Answers2

4

When statisticians say that they prefer parsimonious models they are really cautioning against over-fitting. There really is no other statistical reason to prefer a parsimonious model other than ease of interpretability and, generally speaking, that's a silly reason for a statistician. I think Gelman does a good job articulating this here. Of course, there are conceivable reasons why we'd sacrifice statistical accuracy for interpretability but these are not statistical reasons.

I realize this doesn't really answer the spirit of your question so I'll attempt to do that as well. The "complexity" of a function is an objective thing. I think many people would agree that a function that is very "curvy" is more complex than a less-curvy function. One way to assess the curvy-ness of a function is to look at the derivative and the following is a way to assess the curvy-ness of a function across it's domain,

$$\int f''(x)^2 dx $$

TrynnaDoStat
  • 8,124
  • 2
    +1 for the specific suggestion of "curvy-ness" (I'd probably have called it a measure of nonlinearity myself) as a measure of complexity. However, I don't think interpretability is the only reason to prefer simpler models. For example, if we start - as George Box put it - with the idea that 'all models are wrong', simple wrong models are often more robust in a particular sense (e.g. in time series, in terms of k-step-ahead prediction errors) than complex wrong models. – Glen_b Feb 11 '15 at 22:28
  • Nice answer, thanks. FWIW, I'm actually interested in parsimony in physically-based numerical simulation models - and you can appreciate how parismony becomes very important as soon parameter non-identifiability/equifinality start to become problems. I asked here because I knew I'd get some good answers :) – naught101 Jun 19 '15 at 08:45
3

The number of parameters often turns out not to be a good measure of the complexity of a function or model. There are a number of ways of measuring complexity in different scenarios - one of the simplest is Vapnik–Chervonenkis dimension. The basic idea is to imagine a bunch of points distributed in the xy plane, some labeled + and some labeled -. Can a curve from a particular model class be chosen such that all the + points are above the line, and all the - points are below? The number of points for which you can do this defines how "wiggly" the model class is, and thus how complex it is.

For example, say that you have two models $f_1(x) = a*x^2$ and $f_2(x)=1-cos(a*x)$, each of which have one tunable parameter $a$. For how many points (say placed between -0.5 and 0.5) can we produce any +/- labeling, for each model? $f_1(x)$ is very poor by this measure, since it can't even produce all labels for two points (if $y_1/x_1^2>y_2/x_2^2$, then you can never get point 1 below the line and point 2 above). $f_2(x)$, however, actually has infinite VC-dimension, since it can produce any labeling for any number of points (see solution B2b here). So if both $f_1$ and $f_2$ both provide good fits to some data (e.g. we can find an $a$ for each that gives a good fit), we would strongly prefer $f_1$ since it comes from a simpler model class and therefore is more likely to generalize well.

Note that we're talking about complexity of function classes here, not specific functions as you implied in your question - typically "parameters" refers not to the input $x$ of a function, but the coefficients of parts of the model ($a$ in my example).

Chris
  • 1,151