Ranking features in logistic regression

Question

I used Logistic Regression. I have six features, I want to know the important features in this classifier that influence the result more than other features. I used Information Gain but it seems that it doesn't depend on the used classifier. Is there any method to rank the features according to their importance based on specific classifier (like Logistic Regression)? any help would be highly appreciated.

Answer 1

I think the answer you are looking for might be the Boruta algorithm. This is a wrapper method that directly measures the importance of features in an "all relevance" sense and is implemented in an R package, which produces nice plots such as where the importance of any feature is on the y-axis and is compared with a null plotted in blue here. This blog post describes the approach and I would recommend you read it as a very clear intro.

Answer 2

To begin understanding how to rank variables by importance for regression models, you can start with linear regression. A popular approach to rank a variable's importance in a linear regression model is to decompose $R^2$ into contributions attributed to each variable. But variable importance is not straightforward in linear regression due to correlations between variables. Refer to the document describing the PMD method (Feldman, 2005)[3]. Another popular approach is averaging over orderings (LMG, 1980)[2].

There isn't much consensus over how to rank variables for logistic regression. A good overview of this topic is given in [1], it describes adaptations of the linear regression relative importance techniques using Pseudo-$R^2$ for logistic regression.

A list of the popular approaches to rank feature importance in logistic regression models are:

1. Logistic pseudo partial correlation (using Pseudo-$R^2$)
2. Adequacy: the proportion of the full model log‐likelihood that is explainable by each predictor individually
3. Concordance: Indicates a model’s ability to differentiate between the positive and negative response variables. A separate model is constructed for each predictor and the importance score is the predicted probability of true positives based on that predictor alone.
4. Information value: Information values quantify the amount of information about the outcome gained from a predictor. It is based on an analysis of each predictor in turn, without taking into account the other predictors.

References:

1. On Measuring the Relative Importance of Explanatory Variables in a Logistic Regression
2. Relative importance of Linear Regressors in R
3. Relative Importance and Value, Barry Feldman (PMD method)

Answer 3

Don't be alarmed. Logistic Regression (LR) can very much be a classification scheme. LR minimizes the following loss: $$ \mathop {\min }\limits_{{\bf{w}},b} \sum\limits_{i = 1}^n {\log \left( {1 + \exp \left( { - {y_i}{f_{{\bf{w}},b}}({x_i})} \right)} \right) + \lambda {{\left\| {\bf{w}} \right\|}^2}} $$ where the $x_i$ and $y_i$ are the feature vector and target vector for example $i$ from your training set. This function originates from the joint likelihood over all training examples, which explains its probabalistic nature even though we use it for classification. In the equation $\mathbf{w}$ is your weight vector and $b$ your bias. I trust that you know what ${{f_{w,b}}({x_i})}$ is. The last term in the minimization problem is the regularization term, which, among other things, controls the generalization of the model.

Assuming all your $\mathbf{x}$ are normalized, for example by deviding by the magnitude of $\mathbf{x}$, it is quite easy to see which variables are more important: those wich are larger c.f. the others or (on the negative side) smaller c.f. the others. They influence the loss the most.

If you are keen on finding the variables which really are important and in the process don't mind kicking a few out, you can $\ell_1$ regularize your loss function: $$ \mathop {\min }\limits_{{\bf{w}},b} \sum\limits_{i = 1}^n {\log \left( {1 + \exp \left( { - {y_i}{f_{{\bf{w}},b}}({x_i})} \right)} \right) + \lambda \left| {\bf{w}} \right|} $$

The derivatives or the regularizer are quite straightforward, so I will not mention them here. Using this form of regularization and an appropriate $\lambda$ will enforce the less important elements in $\mathbf{w}$ to become zero and the others not.

I hope this helps. Ask if you have any further questions.