5

Akaike information criterion is a measure for goodness of fit of a model that compensates for the number of parameters that were used to build that model.

Consider two linear models, one that is built using person's weight ($w$) and height ($h$): $f_1 = a w + b h + c$. Obviously, this model contains two parameters: w and h. Now, consider a model that uses body mass index (BMI), which is calculated as a combination of weight and height: $f_2 = aB + b$ (here, $B$ stands for BMI). Should BMI in this case be treated as a single parameter vis-a-vis AIC or as two parameters?

What about other complex model parameters that may include non-linear combination of different basic ones?

Boris Gorelik
  • 2,707

3 Answers3

7

I would change "Akaike information criterion is a measure for goodness of fit of a model that compensates for the number of parameters that were used to build that model." into "Akaike information criterion is a measure for goodness-of-fit of a model that compensates for the number of parameters that need to be estimated." The larger the number of parameters, the more AIC penalises the goodness-of-fit.

In your example, BMI requires both height and weight, but only one parameter has to be estimated. Hence, if models $f_1$ and $f_2$ equally fit your data, then according to the Akaike information criteria $f_2$ is preferred to $f_1$. (However, model $f_1$ might be more appropriate to investigate the impact of height and weight individually.)

ocram
  • 21,851
  • +1 BMI forces a very specific and theoretically-driven relationship between height and weight. $y=\beta_0+\beta_1height+\beta_2weight$ is way more flexible (e.g., able to spuriously fit unrelated data) than $y=\beta_0+\beta_1*BMI$. That is, there is a good reason for wanting to penalize the former more than the latter. – gung - Reinstate Monica Dec 25 '11 at 23:54
  • Your answer is entirely correct. However, I would also note that the reason why we want to penalize the number of parameters to be estimated is because the number of parameters directly relates to the variance of the estimator. That is why penalized regression methods have lower d.f.'s than non-penalized methods when the same number of parameters are to be estimated. – charles.y.zheng Dec 26 '11 at 13:57
4

Strictly speaking, you cannot compare the models as they are not nested.

But you could build a new model

$f_3 = aw + bh +cB + \epsilon$

at which points the AIC allows you to compare subsets, trading off parsimonious representation against better fit from including more variables. In a nested model like this, you can even formally test using the $J$-test (included e.g. in R's lmtest package, as well as other commercial regression software).

Dirk Eddelbuettel
  • 8,872
2

I enjoyed both @ocram's and @Dirk's answers. I think a part of this question is about data reduction. Weight and height probably exhibit collinearity and can by such logic be combined into one variable.

Frank Harrell suggests in his "Regression Modeling Strategies" that one can use varclus (SAS or R) to examine the variables:

vc <- varclus(~height + weight + other tested variables, sim='hoeffding')
plot(vc, legend=T)

This gives a plot that might help the evaluation of collinearity. Harrell provides an example where he looks at systolic and diastolic blood pressure. He combines these two into one variable using the mean blood pressure. There are a few chapters where he talks about variable transformation and clustering with various methods but I think the best is when you have a natural transformation such as the weight + height --> BMI.

You usually have to reduce the parameters to minimize the shrinkage before going into testing. The AIC method is very dependent on the degree of shrinkage, it can be estimated from $\frac{LR-p}{LR}$ where LR is the log likelihood ration $\chi^2$ and p the number of parameters in the model. The aim is to keep shrinkage above 0.9. In a binary logistic model a rule of thumb is a 10:1 ratio for number of events to degrees of freedom.

I'm still working my way through Harrell's book but I hope I got the basics right. Please comment if something seems a little off.

Max Gordon
  • 5,926
  • 8
  • 34
  • 52