0

I am trying to generate 95% confidence intervals for the accuracy predictions on the trained data set using the caret package and interface.

Some dummy code:

library(earth)

Set control parameters

Tcontrolparam.Acc = trainControl("cv", savePredictions = TRUE,  classProbs = T, returnResamp = "final")

Set output to factor - classification - dw about names - just dummy data

trees$class = c(rep(c("John","Steve"),nrow(trees)/2),"Steve")

Model 1 using glm model

set.seed(0)
x = train(x = select(trees, -class), y = trees$class,
                                  method = "glm",
                                  metric = "Accuracy",
                                  trControl = Tcontrolparam.Acc)

Model 2 using MARS (earth) model

y = train(x = select(trees, -class), y = trees$class,
          method = "earth",
          metric = "Accuracy",
          trControl = Tcontrolparam.Acc)

Combining models into list and generating resampled results

Model.Objects.List = list(x,y)

Results = resamples(Model.Objects.List)

Summary.Results = summary(Results)

The code above and specifically Summary.Results gives a mean value for accuracy across the resamples but not the confidence intervals. Is there an easy way of computing confidence intervals using caret or externally across many models?

JFG123
  • 133

0 Answers0