1

I would like to compare the classifier performance when there is a newly added variable to the dataset. Say the original prediction was with 10 input variables. The new one is with 11 inputs.

My purpose is to check whether it is practically useful to collect this new input variable. I am particularly interested in comparing the specificity performance due to my setting.

What would be the correct way to statistically test whether it is indeed useful in terms of specificity to collect this new input?

What I did was:

I have approximately 300 data points.

  1. I ran a simple decision tree algorithm with 10-fold cross-validation with stratified sampling for the original dataset and the new dataset. I used the same folds for both.
  2. Then, conducted an unpaired t-test for the specificities based on the folds for each dataset.

I am unsure about step (2). I am not sure if I should do a paired or an unpaired t-test. Or maybe some other test?

I haven't been able to find a source dealing with this yet. There is a similar question here, but there is no specific answer for which statistical test would be appropriate.

mezbaha
  • 11
  • How about you look at the predictive performances of the two models and you determine if the difference is significant enough for you? – user2974951 Jul 03 '23 at 11:30
  • Thanks for the suggestion. I think I understand your point. I guess I could report the % of the folds the new dataset performs better and its magnitude in terms of average and sd. But I believe I should mention this: Ultimately, I want to be able to show that the new dataset's specificity scores are statistically significantly better than the original dataset's. – mezbaha Jul 03 '23 at 12:44

0 Answers0