What do I make of all classification scores being equal to 1?

Question

I've built an XGBoost classifier with following code:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)
xgbc = XGBClassifier()

xgbc.fit(X_train, y_train)
y_pred = xgbc.predict(X_test)
rep = metrics.classification_report(y_test, y_pred)

print(rep)

And I get the scores as follows.

Then I performed 5 fold cross-validation:

cross_val_score(xgbc, X, y, cv=5)

And got the following cv scores.

• 1.0
• 1.0
• 0.99497487
• 0.98994975
• 0.98492462

I'm fairly new to machine learning, and I can't figure out if my model is pefroming excellently or if it has overfit.

Answer 1

This is not a case of the learner (XGBoost here) not giving results summing up to one. What is returned by cross_val_score are the scores from our learner (xgbc here) for each run of the cross-validation. In general, the performance attained in your test set is exceptionally high, note thought that if we use a dataset for model selection, we want another independent (or at least hold-out) dataset for evaluation. Here as we are using X in its entirety in cross_val_score we do get a somewhat optimistic view of xgbc's performance as at least part of the X has been used to train the model as well as report is performance, at first instance, I would suggest using X_test/y_test. (Comparing different learners as a whole needs to be done via nested CV - see the CV.SE thread on: Nested cross validation for model selection as a starting point).