Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.
Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations and how to optimally select model parameters.
Methods for cross-validation usually involve withholding a random subset of the data during model fitting (training set) and quantifying how accurate the withheld data are predicted (testing set) and repeating this process to get a measure of prediction accuracy. When this partitioning procedure happens once, it's called the holdout method.
The holdout method has two basic drawbacks:
- In problems where we have a sparse dataset we may not be able to afford setting aside a portion of the dataset for testing
- Since it is a single train-and-test experiment, the holdout estimate of error rate may have high variance due to the random nature in which the data was split.
One approach to dealing with these limitations is to use k-fold cross validation.
- Create k equally sized partitions (i.e. folds) of the data. In practice k is often set to 10.
- For each of the k partitions, use k-1 folds for training the model and the kth fold for testing.
- For each of the k experiments, you'll get a prediction error. The average of the k prediction errors is the true error rate.
The advantage of k-fold cross validation is that all the examples in the dataset are eventually used for both training and testing, so it matters less how the data is partitioned. The variance of the resulting estimate is reduced as k is increased. The disadvantage of this method is that the training algorithm has to be rerun from scratch k times, which means it takes k times as much computation to make an evaluation.
When we set k = n, this is known as leave-one out cross validation, because each partition is made up of n-1 training data and 1 testing.
