0

I am trying to use Binomial Logistic regression to identify features from X-rays, which are associated with a disease state. The idea was to try get Odds Ratios for important radiographic features*. It's pilot work right now, with only X-ray based variables being considered.

Since some features might interact and multiply their effects, I used stepwise Logistic regression to select the 'best' possible model, as defined by BIC.

My sample is not huge... 45 controls to 28 cases.

When looking at significant features in isolation (one IV logistic models), the odds ratios are reasonable looking:

Feature      OR      Lower CI  Upper CI
AC_6       4.6186    2.1386    9.9745 
SC_6       3.0416    1.5989    5.7862
SC_2       2.7387    1.4714    5.0976
TC_10      1.7693    1.0285    3.0437

However, the 'best' model (with a BIC of 50, 'worst' has model BIC:100) sees the Odds ratios and confidence intervals jump up - quite a lot.

Feature     OR   Lower CI  Upper CI
const   0.5544    0.1849    1.6624
AC_6   26.4084*   3.7665  185.1573*
SC_2    9.0166    1.8901   43.0129
TC_10  16.9434*   2.0030  143.3262*
SC_6   10.1433    1.8199   56.5358

The idea of using the multivariate model is quite attractive; the features that were selected fit in nicely with current literature concerning the disease. Together, they paint a plausible and interesting picture of how this disease might develop.

However, I feel I need some help interpreting these results.

  1. Firstly, I am trying to diagnose why this happened.
  2. Secondly, I am trying work out what conclusions can safely be made from these results

My current ideas on why this happened are:

  1. The sample size is too small for a 4 variable model. Asking too much with too little data, and precision is suffering.

  2. Very large Odds ratios; therefore larger hikes in the uncertainty

  3. Sparsity of outcome: Some of my features are concentrated around 0. No samples actually have a value of 0, but many sit at ~0.1. However, most features are on a scale range between -2 and +2. Zero meaning the feature is absent. -2 and +2 representing extremes of a range of morphologies (imagine knee flexion; at -2 the knee is fully flexed, at 0 its neutral, at +2 its hyper-extended). Here is an example of the distribution between cases and controls in one X-ray feature: Example distribution of an X-ray feature

  4. Software issues (as discussed in other posts)

Can anyone advise on how I might narrow into a main cause?

As to what conclusions can be made:

  1. None from the multivariate analysis - its just too little data. Focus on the singe IV models
  2. The combination of X-ray features could represent real world phenomenon, but you can never report these, as you will be laughed out of any conference...
  3. Try remake the models smaller and use BIC + more acceptable confidence intervals as the criteria

Any advice would be appreciated! *I hope this doesn't come across as lazy repetition of a frequently asked question! have read through previous posts on wide confidence intervals - however I am struggling to pin down which explanations might work best for this problem. I am keen to learn how one might narrow the possibilities and make the correct inferences.

Maks Hall
  • 41
  • 1
    Try searching this site for separation or quasi-separation and see if that helps. – mdewey Oct 23 '22 at 14:08
  • Some near-dups: https://stats.stackexchange.com/questions/275058/very-wide-confidence-intervals-for-odds-ratios, https://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression, https://stats.stackexchange.com/questions/165068/continuous-variable-has-very-large-odds-ratio-in-binary-logistic-regression Do not use stepwise, try regularization. – kjetil b halvorsen Oct 24 '22 at 13:22
  • Thank you both @mdewey @kjetil! From doing more reading - it does sound like quasi separation. Which unfortunately python Statsmodels gives no warning for. I am struggling to understand whether regularization allows the traditional interpretation of the odds ratios though? I just want to get odds ratios for my combined selected features. However seems like cheating that I can pick the penalty and make things 'look' respectable again? – Maks Hall Oct 24 '22 at 16:12
  • 1
    The most highly voted answer (not the accepted one) in https://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression suggested by @kjetilbhalvorsen has a host of other ideas apart from regularisation. – mdewey Oct 24 '22 at 16:18
  • Thank you @mdewey, this is very helpful. I understand regularization is not valid when trying to use the coefficients for inference: https://stats.stackexchange.com/questions/428349/get-odds-ratios-with-confidence-intervals-from-a-lasso-regression-model. So good to have more options to look at! – Maks Hall Oct 24 '22 at 16:28
  • If anyone else reads this - this is a nice text on the subject and a good reference for your report! https://academic.oup.com/aje/article/187/4/864/4084405 – Maks Hall Oct 26 '22 at 16:23

0 Answers0