Comparing non-nested Cox models: AIC & Partial likelihood test?

Question

I'm doing survival Cox PH analyses and want to understand how an important continuous variable impacts time-to-event (model1).

This important variable is frequently dichotomized in my research field (model2). I'm studying how a new cutpoint (model3) would perform.

When building a Cox model, anova(modABC, modABCD, test = 'LRT') is pretty straightforward to use when comparing nested models.

But what about non-nested Cox models? If I would like to see how continuous (model1) compares to conventional binary (model2) or to new cutpoint (model3)?

AIC seems like the most logical, as common pratice and shown here.

But for instance, is merely having a smaller AIC value sufficient for stateing - considering proper power (n of events), careful assumptions checking - model3 seems a better choice than model2?

> extractAIC(model1)
[1]    8.000 2635.941
> extractAIC(model2)
[1]    8.000 2640.818
> extractAIC(model3)
[1]    8.000 2638.635

I checked this post, pointing to partial likelihood tests, from this 2002 Biometrika paper.

With nonnestcox package it can be implemented:

> plrtest(model3,model2, nested = FALSE)
Variance test 
  H0: Model 1 and Model 2 are indistinguishable 
  H1: Model 1 and Model 2 are distinguishable 
Fine: p = 8.05e-05
Non-nested likelihood ratio test 
  H0: Model fits are equally close to true Model 
  H1A: Model 1 fits better than Model 2 
    z = -0.265,   p = 0.605
  H1B: Model 2 fits better than Model 1 
    z = -0.265,   p = 0.3953
  H1: Model fits not equally close to true Model 
    z = -0.265,   two-sided p = 0.7907

Is this an appropriate way to carry the analysis? I also have trouble understanding Fine: p = 8.05e-05, indicating distinguishable models, but "high" p-values for H1A and H1B. Would H1B be a better fit? I mean, I know variance is a different test than partial likelihood.

Thanks a lot! I'm a new R user. Also running other regression modelings, but this Cox part is important to the big picture. Also know dichotomous approach for continuous variable is not stats gold standard.

I'm also solving the same question and I'm a bit stuck with why variance test is so significative but none model seem to follow the H1 in the likelihood test... what's the interpretation of this? Is this saying that any model is relevant? — garcesj, Dec 15 '21 at 15:27
I faced this issue for a while and couldn't make much sense of this. I ended up moving from this approach. Without showing a model test p-value, results were already pretty clear to the point of the story (three bins better than two, as a starting discussion towards more continuous approaches). — Titorelli, Dec 17 '21 at 10:01

score 1 · Accepted Answer · answered Aug 28 '20 at 14:27

With respect to AIC, the Wikipedia page says:

AIC can be used to form a foundation of statistics that is distinct from both frequentism and Bayesianism.

Convincing a reader that you can use the AIC to choose among non-nested models depends on the reader's appreciating what AIC means.* Someone with that level of statistical sophistication will probably already be convinced that a search for a "better" cutoff of a continuous predictor is not worth the effort. I'm struck that even the simple linear form for the continuous predictor used for model1 seems to outperform both cutoffs of model2 and model3 in terms of the AIC.

Such a reader would certainly appreciate a more sophisticated modeling of the continuous predictor. For example, with a spline fit of that predictor you could use standard tests of nested models to come up with a useful nonlinear relationship. Maybe such modeling will find a value of the predictor at which the hazard changes rapidly, which might be used to argue for a cutoff, but even then the continuous model should contain more information. A plot of the continuous relationship of the predictor to outcome should be highly convincing even to those who would prefer to believe in cutoffs.

With respect to the Fine test, I don't see a discrepancy in the results (although I'm not an expert on that). The Fine test tries to determine which of a set of models might be closest to a "true" model. As I understand it, you can tell that your model2 and model3 differ from each other (the first reported test), but the Fine test can't determine that either of them is closer to the "true" model than the other (the second set of tests). Maybe they are in some sense on different "sides" of the "true" model yet equally far away from it.

*For reference, with nested models that might also be evaluated by chi-square tests, the AIC criterion is equivalent to a p-value cutoff of 0.157 versus the traditional value of 0.05. Also, as discussed on the page you linked, not all are convinced that the AIC is suitable for selecting among non-nested models.

Comparing non-nested Cox models: AIC & Partial likelihood test?

1 Answers1

Linked