25

I'm brand new to this R thing but am unsure which model to select.

  1. I did a stepwise forward regression selecting each variable based on the lowest AIC. I came up with 3 models that I'm unsure which is the "best".

    Model 1: Var1 (p=0.03) AIC=14.978
Model 2: Var1 (p=0.09) + Var2 (p=0.199) AIC = 12.543
Model 3: Var1 (p=0.04) + Var2 (p=0.04) + Var3 (p=0.06) AIC= -17.09

    I'm inclined to go with Model #3 because it has the lowest AIC (I heard negative is ok) and the p-values are still rather low.

    I've ran 8 variables as predictors of Hatchling Mass and found that these three variables are the best predictors.

  2. My next forward stepwise I choose Model 2 because even though the AIC was slightly larger the p values were all smaller. Do you agree this is the best?

    Model 1: Var1 (p=0.321) + Var2 (p=0.162) + Var3 (p=0.163) + Var4 (p=0.222)  AIC = 25.63
Model 2: Var1 (p=0.131) + Var2 (p=0.009) + Var3 (p=0.0056)                  AIC = 26.518
Model 3: Var1 (p=0.258) + Var2 (p=0.0254)                                   AIC = 36.905

thanks!

MEL
  • 285
  • Could you tell us the difference between (1) and (2)? Clearly something changed, because Model 3 in (1) and Model 2 in (2) are nominally identical but the p-values and AIC differ. – whuber Apr 04 '11 at 18:52
  • 2
    This question has been reposted two times, which means that not only we have to close them, but also the associated response(s) that were already provided to you. Could you please register your account (see the FAQ), and pay attention to StackExchange posting policy in the future? Thanks. – chl Apr 04 '11 at 20:56
  • @whuber,

    I'm afraid I don't understand your question fully. It's probably my lack of statistical understanding. But to try to clarify. Model 1 has 4 variables, Model 2 has 3 variables and Model 3 has 2 variables. The variables are in the same order in every model (meaning variable one = temp in each model).

    I think @GaBorgulya and @djma answered my question perfectly. Variable 4 IS correlated with variable 3. AH-HA! Makes sense.

    thanks oodles!

     – MEL Apr 07 '11 at 11:23
  • I've converted your response to the above comment. If you feel one of the current responses helped you or answered your question, don't forget to accept it, as kindly reminded by @richiemorrisroe. BTW, good to see you registered your account. – chl Apr 08 '11 at 01:01

3 Answers3

36

Looking at individual p-values can be misleading. If you have variables that are collinear (have high correlation), you will get big p-values. This does not mean the variables are useless.

As a quick rule of thumb, selecting your model with the AIC criteria is better than looking at p-values.

One reason one might not select the model with the lowest AIC is when your variable to datapoint ratio is large.

Note that model selection and prediction accuracy are somewhat distinct problems. If your goal is to get accurate predictions, I'd suggest cross-validating your model by separating your data in a training and testing set.

A paper on variable selection: Stochastic Stepwise Ensembles for Variable Selection

User1865345
  • 8,202
djma
  • 779
  • 5
    If your goal is prediction accuracy, you want to use AIC (as it minimises the expected KL divergence between the fitted model and the truth). If you want a consistent model selection procedure (fixed p, growing n), you may use, say, BIC instead. Using p-values in stepwise regression to select hypotheses is definitelly not recommended. – emakalic Apr 04 '11 at 23:54
  • 9
    @emakalic - just a quick note, that AIC & BIC are basically just ways of choosing which p-value to use, rather than doing something "fundamentally" different. For AIC, we have a p-value of $0.154$ and for BIC we have a p-value equivalent to $|t|>\sqrt{\log(N)}$. – probabilityislogic Nov 04 '11 at 09:51
  • That's nice to know. I have a question wrt to using the p-value equivalent of BIC in a step-wise regression. Would this be a static value I. E, do I use the initial p-value set for N - k - 1 degrees of freedom. Or does this get updated dynamically at each step, with the addition or subtraction of the degrees of freedom? Does it even matter? – Linus Fernandes May 03 '20 at 20:45
  • @probabilityislogic, can you give any reference that demonstrates what you say regarding the p-value of 0.154? I am having trouble seeing this ... – koteletje Sep 22 '20 at 10:01
  • 2
    @koteletje - Bayesian model selection in social research by Adrian raftery. social methodology, vol25 (1995), 111-163 equation (20) has the connection with the lr chi square statistic – probabilityislogic Sep 23 '20 at 10:43
  • thank you very much @probabilityislogic – koteletje Sep 23 '20 at 11:33
  • See also this question for details on the association between AIC and p-value of a likelihood-ratio test: https://stats.stackexchange.com/q/89214/237402 – retodomax May 05 '22 at 12:44
25

AIC is a goodness of fit measure that favours smaller residual error in the model, but penalises for including further predictors and helps avoiding overfitting. In your second set of models model 1 (the one with the lowest AIC) may perform best when used for prediction outside your dataset. A possible explanation why adding Var4 to model 2 results in a lower AIC, but higher p values is that Var4 is somewhat correlated with Var1, 2 and 3. The interpretation of model 2 is thus easier.

GaBorgulya
  • 3,363
-1

AIC is motivated by the estimation of the generalization error (like Mallow's CP, BIC,...). If you want the model for predictions, better use one of these criteria. If you want your model for explaining a phenomenon, use p-values.

Also, see here.

Community
  • 1
JohnRos
  • 5,684