I've tested out various feature selection methods, such as the F-test, Mutual Information and the Extra Tree (Extra Randomised) Forest Classifier (ETC) as well as PCA (which is technically a feature extraction method), with ETC being used solely for feature selection rather than as a classifier, and have subsequently performed 10-fold cross validation of my models with a combination of GridSearch and pipeline from the wonderful scikit-learn Python package, with these models being random forests, SVM, KNN and logistic regression.
Upon doing this, I found that the average AUC score for the validation sets was the highest for the models when using Extra Tree Classifier as the feature selector, with SVM performing particularly well, and the rest okay, except for logistic regression, which blatantly underperformed compared to the other models, with an average AUC for logistic regression and ETC combined of 0.4761. Weirdly enough, the best performing permutation that was captured upon doing GridSearchCV involved using $L1$ penalty, $C = 0.1$, and selected features of $n = 20$. Since thousands of permutations were performed for each model, this means that some validation AUC scores were in the range of 0.4 and 0.3, which is very unusual given that most sources online state that $0.5\leq AUC \leq 1$, which makes sense.
However, other sources do state than when $AUC<0.5$ this is due to a classification error made by the machine when executing the algorithm, and that one straightforward way to overcome this issue is by doing $1-AUC,$ whilst others state that $AUC<0.5$ indicates that the classifier model is worse than one which classifies completely at random, and this is where my confusion arises. So far I have taken the heuristic approach of subtracting the AUC from 1, but I'm very skeptical of doing this since it might be too heuristic for it to be effective. My current code for logistic regression looks as follows
def logistic(data, outcome):
X_test, y_test = data, outcome
pipe = Pipeline([('a', RFE(ExtraTreesClassifier(n_estimators=400),20,step=1000)),('b',LogisticRegression(C=100))])
pipe.fit(X_train, y_train)
auc_score = roc_auc_score(y_test, pipe.predict_proba(X_test)[:,1]))
if auc_score < 0.5:
fpr_svc, tpr_svc, _ = roc_curve(y_test, pipe.predict_proba(X_test)[:,1], pos_label=0)
auc_score = 1 - auc_score
else:
fpr_svc, tpr_svc, _ = roc_curve(y_test, pipe.predict_proba(X_test)[:,1])
print("Test set AUC: {:.3f}".format(auc_score))
plt.plot(fpr_svc, tpr_svc, label='ROC Curve', color='cyan')
plt.plot([0,1], [0,1], color='black', linestyle='--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.show()
default_prob = pipe.predict_proba(X_test)[:,1]
confusion_mat = confusion_matrix(y_test, pipe.predict(X_test))
results = classification_report(y_test, pipe.predict(X_test))
print(results)
f,ax=plt.subplots(figsize=(7,6))
sn.heatmap(confusion_mat,ax=ax,annot=True)
plt.show()
return default_prob, confusion_mat
As it can be seen, I have created a simple if-statement where I subtract the AUC from 1 if it is less than 0.5, and this was done for the plot of the ROC curve too, since I would be getting an inverse (or convex) ROC curve rather than a concave one. Before doing this, when inputting my test data into the function I would occasionally yield a test AUC score greater than 0.5, which resulted in a normal concave ROC curve, but mainly they were around 0.4 or as low as 0.3.
The figure on the left corresponds to an AUC score of 0.629, whilst the one on the right corresponds to an AUC score of 0.401.
Therefore, does anybody know what could be the cause of such volatile and unusually low AUC scores for the combination of ETC and logistic regression? From what I've read online, ETC has the tendency of capturing very intricate highly non-linear relationships among variables, which might explain why logistic regression, which is naturally a linear model (?), underperforms compared to the rest. If it helps, my confusion matrix is the following:
I would highly appreciate any sort of input or help by anyone.
