3

In the statistical courses I've taken, which are mostly introductory, when I have a model I would make hypotesis tests to reduce it to the simplest form and am effectively done. It is my understanding that this is not the case with predictive models since the goal isn't model reduction but predictiveness of the model based on metrics like R^2, Brier, ROC etc.

In my particular situation I want to fit a logistic regression on my data set. If I go by the rule of thumb of 15 obs. in the least frequent outcome per predictor then I have about 3 times the predictors than I should have to not overfit it. So I quite obviously need to reduce the number of predictors in my model.

So the question is, how does one proceed from the full model? Form what I've read that are three approaches(are there more?): feature selection, data(dimension?) reduction and shrinkage. How does one choose between them?

ChuckP
  • 903
  • These concepts are all related -- shrinkage methods (more specifically L1 regularization) are one form of feature selection and dimension reduction because it reduces coefficients in the linear model toward zero (and can pin some values at 0 in the case of L1/elastic net), i.e. discards some columns. Feature selection can vary by domain - in imaging, it can mean taking the raw image and finding edges. Dimension reduction can also include an unsupervised step like PCA to reduce the size of a matrix, or a rotation method like ICA. What it means to "choose" one depends on definitions and goals. – Sycorax May 23 '16 at 14:32

1 Answers1

2
  1. Decide on some quality measure you wish to optimize. In your specific case of logistic regression, this should include some trade-off between Type I and II errors. Are false positives worse than false negatives?
  2. Then run cross-validations to compare the different ways of reducing your model complexity. Choose between feature selection, dimension reduction and regularisation depending on which approach yields the best model, as per the quality measure defined above.

And if you are willing to go beyond logistic regression, there are of course a couple of machine learning approaches, like random forests, that are "somewhat" orthogonal to your three approaches, although RFs in particular include elements of feature selection.

Stephan Kolassa
  • 123,354
  • My data is medical and the positive response is death so Sensitivity is probably more important. Which implies that ROC, Brier would not be a good measure? However to find sensitivity for Logistic regression one would have to specify a cut-off value, am I wrong to assume it is only appropriate for that particular test set? or can you pool the cut-off value somehow? – ChuckP May 23 '16 at 15:20
  • 1
    Many of the questions in your comment are answered in our archives. Please use the search feature. If your question has not been answered, or you have further questions, feel free to ask a new question. – Sycorax May 23 '16 at 15:24
  • I've tried to search for it but can not find the answer that I'm looking for. I can't wrap my mind around what it means for a model to have a higher sensitivity than another model in a probability setting. If I use the same cut-off for both models, one will have higher sensitivity at that cut-off, but for some other cut-off the other may be higher? Another question is, for a model with a tuning parameter, say lasso, should I use the same quality measure to tune the parameter? – ChuckP May 23 '16 at 22:58
  • You are asking a couple of interesting questions, but (a) they go beyond the original question, and (b) there are multiple ones. Could you please write up separate new CV questions on these and post them? They may be closed as duplicates as @GeneralAbrial writes, but then you will at least have answers. Plus, they will be good for future generations to search for. – Stephan Kolassa May 24 '16 at 09:59