Does a threshold effect the training or testing fold in cross validation?

Question

I am trying to better understand how changing a threshold affects a cross validation model. So if you trained a random forest model, the default threshold is threshold=0.5. And I understand that if the predicted result gets a score >0.5 it is considered a positive case and vice versa. But if you have a 5-fold cross validation model, is the model looking at what happens in the first four folds, and then looks at the threshold in order to give you the results on the test fold OR does it apply the threshold on just the test fold? Or in other words, what does the threshold change? The results of the training folds or just of the testing fold?

I guess more technically speaking, looking at the example below, it appears that the results given for each fold is for the testing fold. So does that mean the threshold is evaluated on the testing fold and the training folds make don't care about the threshold?

attach(iris)
#create a binary outcome on Sepal.Length
iris <- iris %>% mutate(Sepal.Length=ifelse(Sepal.Length>5.0,"aff","neg")) 


ctrl <- trainControl(method="cv", 
                     number=5,
                     summaryFunction=twoClassSummary, 
                     classProbs=T,
                     savePredictions = T)

model <- train(Sepal.Length ~ ., data = iris, trControl = ctrl, method= 
                    "rf", preProc=c("center","scale"), metric="ROC",importance=TRUE, tuneGrid = 
                    data.frame(mtry = 2)) 

#examine outcome at every fold
print(model$pred)

> print(model$pred)
    pred obs   aff   neg rowIndex mtry Resample
#1    aff neg 0.616 0.384        7    2    Fold1
#2    neg neg 0.116 0.884       10    2    Fold1
#3    aff aff 0.602 0.398       15    2    Fold1
#4    aff aff 0.894 0.106       19    2    Fold1
#5    aff neg 0.706 0.294       25    2    Fold1
#6    aff neg 0.716 0.284       27    2    Fold1
#7    neg neg 0.020 0.980       43    2    Fold1
#8    neg neg 0.034 0.966       48    2    Fold1
#9    aff aff 1.000 0.000       51    2    Fold1
#10   aff aff 1.000 0.000       60    2    Fold1

Answer 1

But if you have a 5-fold cross validation model, is the model looking at what happens in the first four folds, and then looks at the threshold in order to give you the results on the test fold OR does it apply the threshold on just the test fold?

You train a model based on the first 4 folds, then this model predicts the probabilities for the test fold. With this probability, the test fold are assigned labels and this is checked against the actual labels, with a 0.5 cutoff, to calculate the accuracy, ROC etc.

Or in other words, what does the threshold change? The results of the training folds or just of the testing fold?

The threshold doesn't change. At least not in the example you showed. If you want to change the threshold to make a prediction, its another thing altogether and has nothing to do with cross-validation. This involves re-writing a predict() function that sets the cutoff at something and incorporating this into randomForest() and train() .

It appears that the results given for each fold is for the testing fold.

To add to the above, yes every data point in your data has a chance to be in the test fold and be predicted by the other 4 folds.