When using sequential feature selection approach, say forward feature selection, I want to stop adding new features when the improvement in model scores is smaller than a certain level. Is there a standard criteria being used in this context as a stopping condition?
It reminds me of the case of using the elbow method in order to choose an optimal number of clusters. Are BIC/AIC relevant in the current problem? are there other methods? I also want to consider looking at some average of the last n additions of features, assuming there are some local decreases in scores, followed by improvement in scores in the next features being added (so I don't want to stop after only one feature that doesn't add significant improvement in scores).
I also looked at the scikit-learn implementation for sequential feature selection, and they don't support a stopping criteria which is based on the actual data, but require a predefined number of features to select, which for my understanding can result with either loss of valuable information / overfit...
