0

I am currently trying to choose which parameters to use in my artificial neural network. Because the end goal is a comparison between the logistic regression and the neural network and I have already used the likelihood ratio test for exluding variables on the same dataset for the logistic regression, I thought the results could justify the variable selection for the neural network as well. Especially because a comparison should be as similar as possible. Is my thought process correct, or am I missing out on something?

MauM99
  • 53
  • Welcome to Cross Validated! The likelihood ratio testing you’re doing to select variables in your logistic regression is at least evocative of stepwise regression, and “Stepwise regression model-building is…pants., according to our Dr. Alexis Dinno. Perhaps you can go into why you want to do any of this analysis. – Dave Aug 29 '22 at 17:31
  • this is going to be tough because of the flexibility of neural net params and the fact that the LRT doesn't "regularize". More common in such flexible models is to use Cross Validation, which this website is named after. – John Madden Aug 29 '22 at 17:32
  • @ Dave The comparison is part of my bachelors thesis. My end goal is to compare the methods with different indicators such as the AUC or the H measure regarding the topic of credit scoring. For the logistic regression I was have not excluded the variables stepwise, but rather used the whole dataset (only to exlcude, not to build a model). As far as I know the LR test shows whether removing a variable has a significant impact, therefore, and because I used the whole set to calculate the values, I thought could used it here as well. – MauM99 Aug 29 '22 at 17:53
  • @JohnMadden In which way should I used a Cross Validation for parameter selection? – MauM99 Aug 29 '22 at 17:55
  • @MauM99 (this approach requires training the neural net a bunch of times, if that takes too long we have to think a little harder). Compute out of sample error of the neural net trained with and without a given variable via CV, ideally several times to get a distribution of these, and compare them. Lower is better. – John Madden Aug 29 '22 at 17:58
  • @JohnMadden Would I have to do it just once for every variable or should it be done stepwise? And wouldnt I need to have the final network architecture beforehand or does the combination of architecture and parameters have no impact? – MauM99 Aug 29 '22 at 18:04
  • @MauM99 Stepwise would do it, if that is computationally feasible. Yes, choosing neural net architecture is hard too, and will have an impact (on this as it would the LRT). – John Madden Aug 29 '22 at 20:39
  • @JohnMadden I have one more question that came up: Since I the results are binary (default or no default), what indicator should I use to compare the out of sample error. MSE an the like aren't useful here, should I use the brier score? – MauM99 Aug 30 '22 at 21:00
  • Let us continue this discussion in chat. – John Madden Aug 30 '22 at 21:14

1 Answers1

0

If you are using a neural network, I recommend that you don't do feature selection. The usual approach is to just include all the available features, and let the neural network learn which ones are important.

This assumes you have a very large training set. If you don't have a large training set, a neural network might not be the best model.

D.W.
  • 6,668