I am creating a couple of models (RF, SVM, LR) and I want to evaluate all of them on a certain PPV (0.7).
This question and this question helped me write my code:
#Run RF
data_ctrl_null <- trainControl(method="cv", number = 10, classProbs = TRUE, summaryFunction=twoClassSummary, savePredictions=T, sampling=NULL)
rf_model <- train(outcome ~ ., data=htn_df, ntree = 2000, tuneGrid = data.frame(mtry = 69),
trControl = data_ctrl_null, method= "rf", preProc=c("center","scale"), metric="AUC",importance=TRUE)
#Create an ROC
myRoc <- roc(predictor = rf_model$pred$affirmatory, response = rf_model$pred$obs, positive = 'affirmatory')
#Find threshold for PPV=0.7
coordinates <- coords(myRoc, x = "all", input = "threshold", ret = c("threshold", "ppv"))
plot(t(coordinates))
But the problem there is no threshold the allows me to get a PPV =0.7; the lowest my PPV goes is 0.7417722 at threshold=Inf and the next lowest PPV is 0.7436548 at threshold= 0.7915000. Below is my plot.
Can someone explain to me conceptually what is going on?
EDIT
From user Calimo's explanation below, I determined that the issue is because of the way my code is interpreting positive and negative cases (should be positive=affirmatory and negative=negatory, or control), which is not what I'm seeing.
myRoc
Call:
roc.default(response = rf_model$pred$obs, predictor =
rf_model$pred$affirmatory)
Data: rf_model$pred$affirmatory in 102 controls (rf_model$pred$obs
affirmatory) > 293 cases (rf_model$pred$obs negatory).
Area under the curve: 0.9008
Saw that the pROC package actually allows you to set cases and controls. I had to modify myRoc:
myRoc_new <- roc(controls =
rf_model$pred$affirmatory[rf_model$pred$obs=="negatory"],
cases=rf_model$pred$affirmatory[rf_model$pred$obs=="affirmatory"])
Now I can see that my model supports a PPV from 0.25-1 at various thresholds. However, the coordinates only provides me with near thresholds i.e. threshold= 0.4277500 gives PPV= 0.6981132 and threshold= 0.4312500 gives PPV= 0.7047619. This is so close to PPV=0.7 but can I get the exact threshold?

positive = 'affirmatory'argument to roc will be ignored unless I'm very mistaken. – Calimo May 13 '20 at 07:06