1

So essentially I have patient data for a period of 5 years: same patient each year each person has 5 predicted risks (for each year) - e.g. in 2017 there risk was 2%, in 2018 the same patients risk was 8% etc The equation predicts the risk of an event in 5 years n is approx 100

All data inputs has been converted to numerical data I want to asses how good the equation was at predicting the outcome. The outcome is already known. Note: there are no false positives as the population being analysed all eventually developed the condition. This is due to the population, not because anyone was excluded. Would an AUCROC case be appropriate for this? Are there any other statistical tests available (other than sensitivity and specificity)

If I were to use AUCROC analysis, would it be appropriate to make 5 aucroc curves for each year?

edit: It is a prognostic predictive model that provides the risk of developing a condition in 5 years - its input is continuos variables. I seek to externally validate it

Thank you guys, Id appreciate any help

RazaS
  • 11
  • Please edit the question to explain what you mean by 'there are no false positives as the population being assessed has the binary outcome of "Yes".' Does that mean that all in the model eventually developed the event in question? Did you exclude from the model individuals who didn't develop the event? It would also help if you could show the nature and form of your model, as that might affect the best way to evaluate it. Please add that information by editing the question, as comments are easy to overlook and can be deleted. – EdM Dec 22 '22 at 21:57
  • @EdM updated. Thank you – RazaS Dec 22 '22 at 22:45

1 Answers1

0

The area under the receiver operating characteristic curve (AUROC) for each year is a good start at validation, but that doesn't provide information about the calibration of your model: how closely predicted and observed event probabilities agree. For that you need to set up a formal calibration curve.

In survival analysis like this (this seems to be a discrete-time survival model, but without censoring) you start by choosing a time for comparison. In your situation you presumably want to choose all 5 times of interest, which you would examine separately.

The simplest approach is to group cases by their predicted probabilities of having the event at the time of interest. Then plot the observed probability of events within each group against the predicted probability for the group. Ideally, you should get a straight line with a slope of 1 for the calibration curve thus produced. There is a way to do this without grouping, estimating the "observed" probability instead with a very flexible fit of event times as a function of covariate values and their interactions. See this page for an overview and links.

This approach can clearly be used for external validation. You should be aware that it can be better to build a model with all of the data. Frank Harrell discusses the issues here. This type of approach can also be used for internal validation, for example by combining it with bootstrapping.

This page might be useful for further study; it's focused on Cox models, but many of the principles apply in general. A site search on "calibrat* survival model" will show several dozen additional pages related to these issues.

EdM
  • 92,183
  • 10
  • 92
  • 267