For the selection of predictors in multivariate linear regression with $p$ suitable predictors, what methods are available to find an 'optimal' subset of the predictors without explicitly testing all $2^p$ subsets? In 'Applied Survival Analysis,' Hosmer & Lemeshow make reference to Kuk's method, but I cannot find the original paper. Can anyone describe this method, or, even better, a more modern technique? One may assume normally distributed errors.
Asked
Active
Viewed 2,394 times
3 Answers
13
I've never heard of Kuk's method, but the hot topic these days is L1 minimisation. The rationale being that if you use a penalty term of the absolute value of the regression coefficients, the unimportant ones should go to zero.
These techniques have some funny names: Lasso, LARS, Dantzig selector. You can read the papers, but a good place to start is with Elements of Statistical Learning, Chapter 3.
Simon Byrne
- 3,486
- 19
- 31
-
2BTW, the penalized R package (http://j.mp/bdQ0Rp) includes l1/l2 penalized estimation for Generalized Linear and Cox models. – chl Aug 25 '10 at 20:07
-
-
LARS is great, BTW. very cool stuff. not sure how I can jam it into the framework of Cox Proportional Hazards model, tho... – shabbychef Aug 26 '10 at 16:56
-
2The Glmnet software has a lasso'd Cox PH model: http://cran.r-project.org/web/packages/glmnet/index.html there is also a MATLAB version (not sure if it does a cox model though): http://www-stat.stanford.edu/~tibs/glmnet-matlab/ – Simon Byrne Aug 27 '10 at 09:29
4
This is a huge topic. As previously mentioned, Hastie, Tibshirani and Friedman give a good intro in Ch3 of Elements of Statistical Learning.
A few points. 1) What do you mean by "best" or "optimal"? What is best in one sense may not be best in another. Two common criteria are predictive accuracy (predicting the outcome variable) and producing unbiased estimators of the coefficients. Some methods, such as Lasso & Ridge Regression inevitably produce biased coefficient estimators.
2) The phrase "best subsets" itself can be used in two separate senses. Generally to refer to the best subset among all predictors which optimises some model building criteria. More specifically it can refer to Furnival and Wilson's efficient algorithm for finding that subset among moderate (~50) numbers of linear predictors (Regressions by Leaps and Bounds. Technometrics, Vol. 16, No. 4 (Nov., 1974), pp. 499-51)
Thylacoleo
- 5,399
- 5
- 26
- 33
-
- yes, the question is somewhat ambiguous; there are, as you mention, many definitions of 'optimal': via information criterion, cross validation, etc. Most of the heuristic approaches I have seen to the problem proceed by stepwise predictor addition/removal: single pass forward addition or subtraction, etc. However, Hosmer & Lemeshow make reference to this method (a variant of work by Lawless & Singhal), which somehow 'magically' selects predictors by a single computation of an MLR (modulo some other stuff). I am very curious about this method...
– shabbychef Aug 26 '10 at 16:55
0
What I learned it that firstly use Best Subsets Approach as a screening tool, then the stepwise selection procedures can help you finally decide which models might be best subset models (at this time the number of those models is pretty small to handle). If one of the models meets the model conditions, does a good job of summarizing the trend in the data, and most importantly allows you to answer your research question, then congrats your job is done.
gung - Reinstate Monica
- 145,122
-
1I think you may be misremembering this. Best subsets is much more computationally expensive than stepwise, but would necessarily catch anything stepwise would, so you would use stepwise to screen & best subsets after. FWIW, I disagree w/ the naive use of these strategies, for reasons I discuss in my answer here: algorithms for automatic model selection. – gung - Reinstate Monica Jul 22 '14 at 17:12
penalizedR package), http://j.mp/cooIT3. Maybe this one too, http://j.mp/bkDQUj. Cheers – chl Sep 05 '10 at 08:29