Validation supervised model

Question

This is my first approact to Machine Learning. I want to know if there are other methods besides train, validation and test.

Thanks.

Answer 1

In the phase of study and analysis of the problem it is possible to test different models of classification. Then the problem arises of how to understand if a model is working better than another and how to choose the best one among those tested. There are numerous methodologies of selection of the model, between which:

1. Leave-One-Out Cross Validation (LOOCV);
2. K-fold Cross Validation (KCV).

The logic Leave-One-Out Cross-Validation is able to overcome the limitations of the previous methodology, in fact as a validation set takes into account only one element of the total dataset. Consequently, it is possible to perform n validation tests, where n represents the numerosity of the total dataset.

In this way at each iteration the model estimates the class of the n-th sample. Then the prediction error is calculated verifying the goodness of the answer provided by the model. The total error of the model is given by an average of the n errors calculated in the samples.

Finally, the K-fold Cross-Validation is basically an evolution of the previous method. The total dataset is divided into k partitions, where k is a reasonable number that allows for a good amount of data within each partition. The method follows an iterative logic in which one partition at a time is used as a test and calculated, accordingly, the prediction error.