Comparing two classifier accuracy results for statistical significance

Question

This is a clearer version of this question.

I have two different cases: one where I have just the final accuracy, and another with the four error types split out:

Case 1 (no error types):

Dataset size: 78,000
Classifier 1: 51% accuracy
Classifier 2: 64% accuracy

Case 2 (with error types):

Dataset size: 78,000
Classifier 1 True Positive:  21%
Classifier 1 True Negative:  30%
Classifier 1 False Positive: 25%
Classifier 1 False Negative: 24%
Classifier 2 True Positive:  15%
Classifier 2 True Negative:  49%
Classifier 2 False Positive: 31%
Classifier 2 False Negative: 5%

In each of these cases, what statistical tests are available to determine whether the differences between the models are significant?

For each of these tests:

What are the assumptions of the test?
What extra information do I need to perform the test?
What is the formula for performing the test?
How should I interpret the results of the test?

Benavoli, Alessio, et al. "Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis." The Journal of Machine Learning Research 18.1 (2017): 2653-2688. — Dave, Mar 16 '22 at 20:12
In case it's not clear for those who are browsing, the paper linked above argues that my question is the wrong question to be asking; I shouldn't be looking for statistical significance / null hypothesis testing. However, I'm going to keep the question up in case it helps those who are looking to do this. — Pro Q, Mar 16 '22 at 20:21
The linked paper does argue for that (though I’m not certain if I agree). However, it has many references about how to do exactly what you ask. — Dave, Nov 15 '22 at 02:18
@Exploring setting a bounty is the usual way on the site to ask for help for answering a question. I was not able to understand the linked paper well enough to write an answer (and neither was anyone else in my university grad lab), so I, too, would appreciate answers, but am not currently willing to pay the rep. — Pro Q, Nov 20 '22 at 21:46

Comparing two classifier accuracy results for statistical significance

0 Answers0