0

I have one group with n=13 and another with n=26 (proportion 1:2).
I have 14 features to use in the classification model.
I am using a logistic regression model.

My questions are:

  1. Is it correct to use a logistic regression with such imbalanced groups?
  2. I know that I have too many features for the number of observations. I read somewhere that I should have 10 or 20 observations for each feature in my model, so I used the leave-one-out cross-validation to select a subset. I organized everything to run every possible combination of features using the leave-one-out CV and found a trade-off of 4 features with 0.69 accuracy. Despite the result not being so exciting, is this approach correct?
  3. Should the final log reg coefficients be estimated as an average of the cross validation coefficients, or should I use the entire dataset to estimate them? (I read somewhere else that I should go with the latter.)

Any help is greatly appreciated!

rfc
  • 1
  • 1
  • 2
    Your data is pretty balanced already. Take a look at this: https://stats.meta.stackexchange.com/q/6349/1352. Also, logistic regression can definitely be used for "unbalanced" data. Your amount of data is cause for major concern. Even with cross-validation, you will likely be overfitting. Don't use accuracy. Finally, a question: do you want to do prediction/classification or statistical inference/null hypothesis significance testing? – Stephan Kolassa Nov 24 '22 at 14:46
  • Thanks so much for the prompt reply, @StephanKolassa.
    I am doing prediction/classification.
    From the links, it seems that there is NO imbalance problem in logistic regression? In this post they suggest that the intercept could be poorly estimated in such imbalaced cases and cause problems when predicting. What do you think?
    I agree that accuracy was a poor choice to describe the problem. I also get sens., specif. and AUC.     – rfc Nov 24 '22 at 15:43
  • It's not only the intercept, but all parameters that will be imprecisely estimated, and of course imprecise parameters can cause problematic predictions (just as in any other model than logistic regression). However, that is just a consequence of having very little information, and there is really very little you can do about that, short of collecting (much) more data. ... – Stephan Kolassa Nov 24 '22 at 15:48
  • ... Specificity and sensitivity suffer from precisely the same problems as accuracy. AUC is slightly better, but best would be probabilistic predictions evaluated using proper scoring rules - but in any case, your data set is so tiny I would not trust any cross-validation to select a "good" model. Best to start with something extremely simple, as guided by domain knowledge. – Stephan Kolassa Nov 24 '22 at 15:49
  • Thanks again. Regarding sample size, the problem is that the data comes from brain surgery, so there is no easy way to increase the observations. Do you suggest to use only a couple features then, instead of this "full search"? Ps: sorry to bother, but is there any reference for the proper scoring rules you mentioned? – rfc Nov 25 '22 at 12:40
  • 1
    You have my heartfelt gratitude for not performing brain surgery just to increase sample size ;-) In that case, I would indeed go with a small set of features, as guided by domain knowledge - in light of your tiny sample size, go with a very small set. Anything more than (say) three will overfit. (You can overfit with cross-validation, too, it's just harder.) Regarding proper scoring rules, the tag wiki contains pointers to literature and links. – Stephan Kolassa Nov 25 '22 at 14:28
  • hahahahah Thank you so much! – rfc Nov 25 '22 at 16:26

0 Answers0