5

Suppose that I have taken 8 machine learning algorithms which are used by researchers most frequently. I have applied these 8 machine learning algorithms over 8 datasets which are publicly available on internet.

I get results like: Random forest works well on 1 dataset. SVM performs better on 2 dataset. How can I conclude which machine learning algorithm among all performs best.

Thanks in advance..

CatsLoveJazz
  • 658
user28681
  • 73

2 Answers2

10

For classification algorithms this would be a good start: Statistical Comparisons of Classifiers over Multiple Data Sets.

To summarize this excellent paper: perform a Friedman test to determine if there is any significant difference between the classifiers and follow-up with an appropriate post-hoc test if there is:

  • to compare all classifiers: Nemenyi test
  • to compare one with all others: Bonferroni-Dunn test

Both post-hoc tests can be visualized neatly in so-called critical difference diagrams.

Marc Claesen
  • 18,401
1

The notion of which Machine Learning algorithm is best is not universal, rather specific to the problem or the dataset you are dealing with.

In case of a single dataset or a problem, apply all learning algorithms and check the performance on out of sample data. Calculate Root mean Square errors between the predicted and actual values of out of sample data, and the algorithm with least RMSE will be the best only for that dataset.

show_stopper
  • 425
  • 1
    Yes, the no free lunch theorem applies. No, it does not mean that you can't generalize anything beyond the data sets you have explicitly tested. Many data sets have a lot in common, even if they stem from entirely different domains. This allows us to generalize the expected performance of learning algorithms to some extent and is why some approaches tend to work well for a lot of practical problems (RF, SVM, NN, ...). – Marc Claesen Oct 29 '14 at 07:23
  • Absolutely true. We can generalize anything beyond the dataset which we have tested only when we have some background knowledge about the current and the new dataset. In case of randomly taken dataset (which I inferred from the question) this does not hold true. Methods like RF which are ensemble tend to work well, as they are very flexible because they use multiple learning algorithms inside them. – show_stopper Oct 29 '14 at 07:30
  • IMHO in order to generalize you need knowledge about both the data and the algorithms. The question is: how well does the heuristic behind the algorithm fit to the struture of the data? E.g. SVM focuses onto few boundary cases, but it will depend on the data (including the application) whether this is a sensible approach (if the single cases are too noisy, you may end up with a margin width that includes most of the cases - and then maybe other heuristics that focus less on single instances, or even give equal weight to all cases from the beginning are better) – cbeleites unhappy with SX Oct 29 '14 at 10:55
  • One more question is there any difference between machine learning and stastitical techniques. I have searched a lot some researchers say that there are some overlap some are saying there is no difference.Can you give some research paper which suggest that both are different from of classification. – user28681 Oct 30 '14 at 11:50
  • statistics vs machine learning is an interesting debate going on for sometimes now.

    In simple words as described in the tread bellow, machine learning is statistics minus any checking of models and assumptions

    http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning

     – show_stopper Oct 31 '14 at 09:51