0

I use 70% of the dataset for training and 30% for testing. I use oversampling on the training dataset with an ANN. I use the test dataset on my ANN and look at the performance of oversampling against not using oversampling. I find the best settings for each ANN using oversampling and for the ANN with no oversampling. Each test setting is tried 10 times.

Now I want to see if there is a statistical significance between the best oversampling model and no oversampling model (when looking at mean performance from 10 tests). Which test should I use to do this?

So basically I have these means from my tests and along with that I calculated the standard deviation from the results.

Test dataset is chosen at random, from two populations (710 and 520 000) each at 30%. Each ANN makes a binary classification and I get the AUC score. How can I see if the ANN's ability to predict this population, based on AUC, is statistically significant?

Don_twice
  • 11
  • What do you need this test for? – Tim May 17 '22 at 17:31
  • To see if oversampling is significant or not – Don_twice May 17 '22 at 18:54
  • What for? Why does it matter? – Tim May 18 '22 at 10:11
  • Now I understand. I am completing a paper as part of my studies. I am allowed to get help and use it as long as I declare it as not mine, which I of course intend to do!

    Based on: https://ieeexplore.ieee.org/document/6790639/authors#authors it seems as if 5x2 cross validation along with a student t-test seems promising? However, I wonder if oversampling might effect this approximation?

     – Don_twice May 18 '22 at 12:21
  • Pardon me for rushing my answer, I read more than the abstract of my prior link. I think my experiment consists of comparing two classifiers and I have enough data for testing. As such I should perform a McNemar's test. However, the testing data needs to be the same for all classifiers tested. This could add errors since I am not using all data for testing. Perhaps perform k-fold cross validation and McNemar's test? Could this be something useful? – Don_twice May 18 '22 at 12:46
  • Cross-validation itself compates the performance, you don't need a hypothesis test here unless there's any reason you need it? See https://stats.stackexchange.com/q/550308/35989 – Tim May 18 '22 at 13:51
  • I understand. We have a good model from prior research and good motivations for data. And it is about comparing the results from using oversampling and not, hence we will want to perform some kind of statistical experiment. I think my prior link points me in the correct direction. – Don_twice May 20 '22 at 15:51

0 Answers0