1

I have developed a cox-model in my research, and I want to compare it to the conventional staging system for prognostic evaluation. The traditional staging system categorizes patients into risk groups with different prognoses. To compare my new model with the conventional model, I have categorized my model prediction into risk groups with the same size (n) as the traditional model. The Kaplan Meier analysis of the traditional and new models shows that both are significant predictors of prognostic outcome ( Log-rank test p<.0.001), but the new model is better for placing patients with poor prognoses at the high-risk groups. My question is: What is the best statistical approach to compare these models?

My solutions so far have been to 1) fit the model group/predictor in a cox-model and compare them with an ANOVA-test (this yields a p-value < 0.001.

  1. perform a comparison of the concordance indexes for the two models predictors: compareC(time,event, new.model.predictor,conventional.model.predictor), concordance index:0.86 vs 0.75 with p-value = 0.02.

I guess approach nr.1 compares the model fit, while approach 2 compares the discriminatory abilities of the two models.

Inputs on which approach would be the most informative and appropriate would be greatly appreciated.

Bent Larsen
  • 101
  • 7

1 Answers1

1

To start, don't destroy the quality of your new model by arbitrarily categorizing its survival predictions. Keep it continuous.

For your ANOVA test, you need to have the models nested. A good way to proceed would be to add the conventional staging as a predictor to your model and then doing ANOVA to compare (your model + conventional staging) against conventional staging. You have to be careful as an anova() function doesn't always do the same test from package to package. You want to do a likelihood-ratio test comparison.

Although useful for evaluating the discrimination capacity of a single model, even Frank Harrell doesn't think that the C-index provides a sensitive test for comparing models. This blog post discusses ways to compare model performance for logistic and survival regression. In particular, the Adequacy Index, calculated from likelihood ratios, provides a measure of "the adequacy of the model that ignores the new predictors."

EdM
  • 92,183
  • 10
  • 92
  • 267
  • Yeah. I see your point, and I am also sort of aware of this sinful practice. The reason for the categorizing was to perform a head-to-head discriminatory comparison with conventional staging which is categorized and also visually illustrate the survival-discrimination-differences in Kaplan-Meier plots which the "general audience" for this kind of paper is comfortable with. But you are of course correct and I think I will pursue your suggestions. Thank you for your high-quality responses. – Bent Larsen Jan 22 '22 at 19:21
  • 2
    @BentLarsen it can sometimes be OK to display stratified results for an audience. Don't do your statistical comparisons on them, however. My suggestion for comparing nested models comes directly from Harrell's linked blog post. – EdM Jan 22 '22 at 19:37
  • If the Cox models are non-nested, you may still be able to perform a non-nested likelihood ratio test ( https://github.com/thomashielscher/nonnestcox ) – carbocation Feb 29 '24 at 15:00
  • 1
    @carbocation see this page for discussion about the Fine test implemented in that package, and the confusing results that can sometimes happen with it. It's best if the comparison can be formulated as a test on nested models, as was possible in the scenario of the question on this page. – EdM Feb 29 '24 at 15:19
  • Thanks. Apologies I may have misread the question here. I thought that the author had applied univariate predictors (which were nested at the time of model design but ultimately yielded single variable predictors) to a new dataset. But you are right: since ANOVA ran without error, I must have misunderstood the question. – carbocation Feb 29 '24 at 19:09
  • @carbocation that's OK. This gave me a chance to review one of my posts from several years ago, which I had completely forgotten about. – EdM Feb 29 '24 at 19:46