In caret what is the real difference between cv and repeatedcv?

Question

This is similar to question Caret re-sampling methods, although that really never answered this part of the question in an agreed upon way.

caret's train function offers cv and repeatedcv. What is the difference in say doing:

MyTrainControl=trainControl(
    method = "cv",
    number=5,
    repeats=5
)

vs

MyTrainControl=trainControl(
   method = "repeatedcv",
   number=5,
   repeats=5
)

I understand cv breaks the set into k-folds (parameter number), and then starts over and runs it parameters repeats number of times.

The only thing I could think of is that maybe regular cv with repeats uses the same exact indexes for the folds each time? essentially running the cv on the same exact folds each time, vs perhaps repeatedcv selects new folds each times?

Can someone clarify?

Answer 1

According to the caret manual(see "reference manual"), the parameter repeats only applies when the method is set to repeatedcv, so no repetition is performed when the method is set to cv. So the difference between both methods is indeed that repeatedcv repeats and cv does not.

---

Aside: Repeating a crossvalidation with exactly the same splitting will yield exactly the same result for every repetition (assuming that the model is trained in a deterministic manner), which is not only inefficient, but also dangerous when it comes to comparing the validation results for different model algorithms in a statistical manner. So be aware of this if you ever have to program a validation yourself.

Answer 2

Admittedly, this is a VERY old post, but based on the code snippets provided by user3466398, the difference is that repeatedcv does exactly that: it repeatedly performs X-fold cross-validation on the training data, i.e. if you specify 5 repeats of 10-fold cross-validation, it will perform 10-fold cross-validation on the training data 5 times, using a different set of folds for each cross-validation.

The rationale for doing this, I presume, is to allow one to have a more accurate and robust accuracy of the cross-validation testing, i.e. one can report the average CV accuracy.

Answer 3

The actual code behind these parameters can be found in the selectByFilter.R and createDataPartition.R (formerly createFolds.R) source files in the `caret/R/' folder of the package.

See these files for e.g. here and here (beware these permalinks may eventually point to older version of the code). For convenience the relevant snippets (as of version 6.0-78 c. Nov 2017) are show below

In selectByFilter.R c. line 157

sbf <- function (x, ...) UseMethod("sbf")
... 

"sbf.default" <-
  function(x, y,
           sbfControl = sbfControl(), ...)
  {
    ...

    if(is.null(sbfControl$index)) sbfControl$index <- switch(
      tolower(sbfControl$method),
      cv = createFolds(y, sbfControl$number, returnTrain = TRUE),
      repeatedcv = createMultiFolds(y, sbfControl$number, sbfControl$repeats),
      loocv = createFolds(y, length(y), returnTrain = TRUE),
      boot =, boot632 = createResample(y, sbfControl$number),
      test = createDataPartition(y, 1, sbfControl$p),
      lgocv = createDataPartition(y, sbfControl$number, sbfControl$p))
...

In createDataPartition.R c. line 227

createMultiFolds <- function(y, k = 10, times = 5) {
  if(class(y)[1] == "Surv") y <- y[,"time"]
  prettyNums <- paste("Rep", gsub(" ", "0", format(1:times)), sep = "")
  for(i in 1:times) {
    tmp <- createFolds(y, k = k, list = TRUE, returnTrain = TRUE)
    names(tmp) <- paste("Fold",
                        gsub(" ", "0", format(seq(along = tmp))),
                        ".",
                        prettyNums[i],
                        sep = "")
    out <- if(i == 1) tmp else c(out, tmp)

  }
  out
}