I understand that doing this leads to data leakage, but if I get better performance on the test set does it really matter?
I tried using caret with sampling = "smote" to use SMOTE on each fold individually, but I just don't have enough samples on the minority class when split into train/test.
I'm only interested in finding features that separate my two classes.
To start, let's point out that using SMOTE on the entire dataset before a test set is split off is bad: as you say, it causes data leakage, and thus generally overly-optimistic estimates of performance. See e.g. SMOTE data balance - before or during Cross-Validation
But here that's not what you've done, and the test set score here should still be an unbiased estimate of future performance of the model. In that sense, based just on what you've done, this is fine.
Applying SMOTE across the training set before using cross-validation for model selection has leaked information to the test folds, but at the end of the day the model that got selected is performing well on the untouched separate test set. Every model that got compared had the same data leakage, so probably the effect was to select a model that performs well in general but also got the best improvements from the data leakage. So it could well be that there's a better model with less improvement from the leakage, but you should've seen that when you retried the experiment with the more-proper approach.
All that said, I suspect the improved performance is just due to noise. If you repeated the experiment with different splits, random states, etc., I expect you'll see the net improvement to be near zero (and maybe negative). (If you have some time to do that sort of experiment, I would love to see it as an answer here.)
First of all, you need to know that the general consensus is turning against SMOTE. It doesn't really solve the problem it is supposed to solve, worse, the actual problem doesn't really exists.
Regarding a general answer: as long the solution is not leaky on the test, you are relatively good to try whatever you want. However you risk having a discrepency betwee, validation and test. What is the point of validation if you don't have consistency with test?
Regarding leakage: SMOTE creates points from the others. For exemple If you have point A and B you can create point (A+B)/2. Then if you split the points you might end up with leakage.
In the first case, using two folds you test the model on A when training on B and test on B when training on A.
In the second case you'll have leaky split typically you'll have following cases:
Yes, it is bad, and it isn't just leakage (+1 to the previous answers). This is because while the synthetic data generated by SMOTE is asymptotically from the same distribution as the data, this is not true for small samples, and imbalanced learning problems almost always have a small sample of the minority class. It is easy to see that this is not the case because the synthetic examples lie within the convex hull of the original data. For a small sample, the convex hull will systematically underestimate the true spread of the distribution. If you gather more real data, some of it will lie outside the convex hull of the original sample. Thus the estimate you get from "test data" containing synthetic data will be biased because the data distribution is not quite the same as that from the true data generating process. Ironically this means that SMOTE itself introduces a bias against the minority class (relative to the theoretical cost-sensitive learning effect implied by the balancing).
"but I just don't have enough samples on the minority class when split into train/test."
that means you don't have enough data for a cross-validation estimate to be stable (low variance) anyway. SMOTE creates more data, it doesn't create more information. SMOTE synthetic data should not be in the test set or in used for evaluation in cross-validation. There really is no benefit to doing so, and lots of potential pitfalls.
BTW, for very small datasets, I tend to use bagging and the out-of-bag performance estimator, rather than cross-validation. You can run it for many iterations to reduce the variance of the estimate and you get and ensemble classifier as a by-product that is likely to work somewhat better,
I understand that doing this leads to data leakage, but if I get better performance on the test set does it really matter?
YES, it matters! The problem with data leakage is that you can no longer trust test set performance. If your test set performance may have "improved" because of the data leakage, then the test set was not actually measuring improvement and should not be interpreted as evidence that one model is better than another.
If I ask my friend to time me while I sprint 100m, and it takes me 100 seconds, but I ask him to write down 10 seconds instead -- that doesn't mean I'm a faster runner, it means my measurements are faulty.
As other answers point out, it sounds like you're doing SMOTE in a way that doesn't actually cause data leakage. If so, this is not a problem (although SMOTE may still be problematic for other reasons). But because of how you opened your question, I wanted to make sure it's clear: data leakage really does mean that you can't trust test "performance."
