5

I want to build a model to predict the outcomes of experiments.

My predictive model gives out scores with an range 1 to 100 values.

I want to test if my predictive scores can be used to classify experimental outcomes as "good" or "bad" groups.

Experimentally, we did the 1000 experiments. Using my predictive model, I have 1000 scores.

To test if my predictive model statistically acceptable, what should I do? I have done ROC and sensitivity test for these 1000 X 2 data.

ROC were plotted for all 1000 experimental data and predictive scores. By looking at the AUC values for the plot (sensitity vs 1-specificity), AUC=0.64.

Let's said if my predictive score has a cut off value of 5, i.e. it is likelihood that the experimental outcome will be "good", score > 5 are likely to have "bad" experimental outcome. I calculate the enrichment of my predictive model, i.e. no. of real "good" results / no. of predictive score < 5.

Did I do anything wrong here?

What else should I do to check the predictive power of a model?

evdstat
  • 191

2 Answers2

7

ROC, sensitivity, specificity, and cutoffs have gotten in the way, unfortunately. Assuming there is nothing between "good" and "bad" and that the success of the experiment was not based on an underlying continuum that should have instead formed the dependent variable, a probability model such as logistic regression would seem to be called for. You may need to do resampling to get an unbiased appraisal of the model's likely future performance. Note that even though a receiver operating characteristic curve is seldom appropriate, its area (also called c-index or concordance probability from the Wilcoxon-Mann-Whitney test) is a good summary measure of pure predictive discrimination. On the other hand, percent classified correctly is an improper scoring rule that, if optimized, will result in a bogus model.

Predicted probabilities are your friend, and they are also self-contained error rates at the point where someone forces you to make a binary decision, if they do.

Frank Harrell
  • 91,879
  • 6
  • 178
  • 397
6

AUC is a good start. You can also calculate what percent of observations were correctly classified, and you can make a confusion matrix.

However, the best single thing you can do is calculate these values using a "test" dataset, who's observations were not used to train the model. This is the only true test of a predictive model.

Zach
  • 23,766