How do survival models "account for censoring"? (Do they?)

Question

Background

I'm teaching an intro stats class in our social / health sciences department and I'm finding myself tripped up on something I'd always taken for granted: namely, the claim that survival analysis methods (from Kaplan-Meier to Cox models) "account for" censored data, and that this is one of these methods' central advantages over other approaches.

(Relevant background: in my research I use, mostly clumsily, some applied statistical and quantitative methods, and I have a good grasp of and intuition for some of the basics. So I'm an okay generalist but I'm not a statistician nor even a statistics grad student by any means.)

The Problem

As I'm preparing some PowerPoint slides for these students, though, I'm realizing that while I have a pretty good grasp on what censoring is in survival methods, I have no idea how something like a Cox model "accounts" for it. I'd never thought about it, and instead assumed that this is just what happened. (It's possible that a professor walked us through this earlier in my education, but it's also possible that I used to struggle in 8am stats lectures.)

Censoring, as you all already know, is the state in which we have some information about individual survival time, but we don't know the survival time exactly (Kleinbaum & Klein, p. 5; I've always liked this plain English explanation). In my experience, sometimes when stats people are trying to sell someone on survival methods, they'll say things like "logistic regression doesn't account for censoring, but Cox regression does!"

There are some great posts on CV about censoring; this is maybe my favorite example. In it, user @Tim gives a terrific explanation of censoring:

Intuitive example of censoring is that you ask your respondents about their age, but record it only up to some value and all the ages above this value, say 60 years, are recorded as "60+". This leads to having precise information for non-censored values and no information about censored values.

I think this is brilliant (in fact I plan on borrowing liberally from it during the lecture, with credit given of course). But it doesn't get really get into how survival analyses actually deal with this, and whether it's a selling point for survival methods ("our methods can do this and yours do not") or just something that pops up when you try and ask survival-type questions ("how long will people live, on average, after being given treatment X?").

The Questions

1. Is it true, strictly speaking, that something like a Kaplan-Meier estimator or a Cox proportional hazards model is "accounting" for censoring?

2. If so, how is it doing that?

3. If a survival model indeed accounts for censoring, is this a "feature" of survival methods over others, or a "bug", an inevitable artifact of the sorts of questions one uses survival methods to answer?

My guesses

Well, not guesses, maybe more like very unclear intuitions I'm not too confident about:

4. Why is censoring a problem? I'm thinking that, if ignored, censoring is a huge potential source of bias in making survival estimates: if you don't know what happened after Mr. Smith dropped out of your study (i.e. was lost to follow-up, i.e. was censored) your estimates may be off in one direction or the other. Maybe he lived a long, long time -- or maybe he died the next day. If this is happening to lots of people in the same way, your estimates may be really, really off.

5. So maybe what survival models have found a way to do is keep in the analysis everyone who contributed survival time, regardless of whether we know about their outcome status, while other methods would simply drop all of those people's data as missing.

Am I way, way off here?

Answer 1

Censoring is built into survival models by incorporating it into the likelihood function underlying the analysis. The most common form of censoring occurs when we observe an item for a finite period of time $T$ and it does not fail in that time. Below I will show you how the censoring is built into the likelihood function and how this affects the Cox proportional hazards model.

---

Incorporating censored data into the likelihood function: As a common example, suppose we have items where the time-to-failure has a survival function $S$ and corresponding density function $f$, both of which area parameterised by some parameter $\theta$. If an item $i$ is observed to fail at time $0 \leqslant t_i \leqslant T$ then it is incorporated into the likelihood function using the density term:

$$f(t_i | \theta).$$

However, if an item $i$ is observed throughout the whole time $T$ and it does not fail then this is considered to be a "right-censored" data point (only known to fail at some time after $G$) and it is incorporated into the likelihoood function using the survival term:

$$S(T|\theta).$$

Suppose we have a survival model based on observation for a fixed period of length $T$, where the times-to-failure for each observation are IID conditional on some underlying parameters. Without further loss of generality, we will have $n$ observed failures at times $t_1,...,t_n$ (all within the interval $[0,T]$) and we will will have $m$ right-censored values that did not fail in the oberved time $T$. The overall likelihood function for this data is then given by:

$$L_\mathbf{t}(\theta) = \bigg( \prod_{i=1}^n f(t_i|\theta) \bigg) \times S(T|\theta)^m.$$

In this likelihood function you can see that the censoring of data is "built in" by the fact that right-censored values are incorporated through their survival function instead of the density function for the time-to-failure.

---

Extending to the Cox proportional hazards model: The Cox proportional hazards model still uses a likelihood function for the observed times-to-failure and survival times, but it now adds covariates to the data and uses an assumption of proportional hazards in how these manifest in the hazard function. This does not change the underlying method of how censored values are built into the likelihood function --- e.g., right-censored values still enter through their survival function instead of the density of the time-to-failure.

---

Extension to other kinds of censoring: The above shows the common case where we have right-censored observations with the same censoring time $T$. Of course, this is not the only kind of censorship that can occur. Another possibility is that we might observe items up to different end-times, in which case the right-censored values would occur with different observation periods $t_{n+1},...,t_{n+m}$. In this case the likelihood function would be generalised to:

$$L_\mathbf{t}(\theta) = \bigg( \prod_{i=1}^n f(t_i|\theta) \bigg) \times \bigg( \prod_{i=1}^m S(t_{n+i}|\theta) \bigg).$$

Another possibility (which is uncommon in survival analysis) is left-censorship, where we know that an item failed no later than some time $T_i$. Left-censored observations enter into the likelihood function through the cumulative distribution function $F$. If we extend our model to assume that we have $r$ left-censored observations with observation times $t_{n+m+i},...,t_{n+m+r}$ then the likelihood function would be further generalised to:

$$L_\mathbf{t}(\theta) = \bigg( \prod_{i=1}^n f(t_i|\theta) \bigg) \times \bigg( \prod_{i=1}^m S(t_{n+i}|\theta) \bigg) \times \bigg( \prod_{i=1}^r F(t_{n+m+i}|\theta) \bigg).$$

And of course, you can extend this event further to allow for more complicated kinds of censorship. In general, if a censored observation is known to fall in some set $\mathscr{A}$ then it should enter into the likelihood function through the probability term:

$$\mathbb{P}(t_i \in \mathscr{A}|\theta) = \int \limits_\mathscr{A} f(t|\theta) \ dt.$$

Answer 2

Some students might benefit from the following way to represent the partial likelihood for a Cox model, as an example of Ben's answer (+1) about censoring in general. Display the partial likelihood under the proportional-hazards assumption (without tied event times) as follows:

$$\prod_{i=1}^{n}\frac{h_0(t_i)\text{exp}(\beta X_i(t_i))}{\sum_{j\in\mathcal{R(t_i)}}h_0(t_i)\text{exp}(\beta X_j(t_i)))}= \prod_{i=1}^{n}\frac{\text{exp}(\beta X_i(t_i))}{\sum_{j\in\mathcal{R(t_i)}}\text{exp}(\beta X_j(t_i)))}$$

where cases with events are indexed by i and their event times are $t_i$, covariate values are $X$ with corresponding regression coefficients $\beta$, and the risk set at time $t_i$, $\mathcal{R(t_i)}$, consists of all cases $j$ at risk of an event at that time. The baseline hazard $h_0(t_i)$ factors out.

The risk score for individual j at time $t_i$, based on that individual's covariate values, is $\text{exp}(\beta X_j(t_i))$. This form emphasizes that the risk for individual i having the event is effectively compared against the summed risks of all individuals at risk at that time.

So the way that the Cox model "accounts for censoring" is by letting all individuals contribute to the partial likelihood, via their risk scores in the denominator, so long as they are at risk of an event. It doesn't matter whether those in the risk set have an event at a later time; they provide information so long as they are at risk and are omitted from analysis thereafter. You can think about that as related to the survival-function contribution that Ben describes. The overall product, however, is only over individuals who have events; that's related to the density-function contribution that Ben describes.

There also can be truncated, as opposed to censored, survival times. That's, for example, how models with time-varying covariate values can be set up: left truncated before the new covariate value takes effect, with right censoring or an event thereafter. Contributions of truncated survival times to likelihood also can be expressed in terms of the density and survival functions; this page presents those contributions as described by Klein and Moeschberger.

One caution: both my and Ben's answers implicitly assume that censoring is non-informative; that is, the fact of censoring provides no information about survival except that survival was longer than the observed time of censoring. That's not always a safe assumption. The review by Leung et al on "Censoring Issues in Survival Analysis," Annu Rev Public Health 18:83–104 (1997) explains and illustrates those further considerations.

Answer 3

I thought I’d directly answer the specific questions above: this is intended to address the top-level conceptual issues with some thoughts on teaching to this kind of audience, rather than a technical explanation.

---

The Questions

1. Is it true, strictly speaking, that something like a Kaplan-Meier estimator or a Cox proportional hazards model is "accounting" for censoring?

Yes!

---

2. If so, how is it doing that?

See other answers for technical details on how implemented for a Cox PH model.

I find the intuition clearer for Kaplan-Meier estimation, and in my experience (short-course teaching of these methods to researchers/analysts in health sciences settings) this is reasonably approachable for the kind of audience you are dealing with, or at least a good starting point.

(I appreciate that this question is as much for your own understanding as being about what/how to teach your students)

This applies for both visualisation (“included in denominator up to point of event/censoring” with x marks the spot on the survival curve) and for calculation (showing how we calculate cumulative survival by multiplying the preceding interval’s cumulative survival with the current interval’s survival in the set of individuals at risk).

---

3. If a survival model indeed accounts for censoring, is this a "feature" of survival methods over others, or a "bug", an inevitable artifact of the sorts of questions one uses survival methods to answer?

Very much by design – these methods were developed to address the problems that arise when survival times are censored (and to maximise efficiency, see below)

---

4. Why is censoring a problem? I'm thinking that, if ignored, censoring is a huge potential source of bias in making survival estimates: if you don't know what happened after Mr. Smith dropped out of your study (i.e. was lost to follow-up, i.e. was censored) your estimates may be off in one direction or the other. [snip]

If your time data are subject to censoring, but you have ignored this in analysis, then results will be biased. In some cases (since there are lots of different ways in which one might inappropriately address this issue) you might be treating people with only short-term follow-up as though they had the full period of follow-up.

Different issues can arise depending on how that inappropriate “non-survival analysis” were to proceed: for example, issues arise if the risk set accrues across a period (e.g. infants born with a particular condition across a ten year study period) and outcomes tend to be in earlier periods (e.g. highest risk in first two years of life).

---

5. So maybe what survival models have found a way to do is keep in the analysis everyone who contributed survival time, regardless of whether we know about their outcome status, while other methods would simply drop all of those people's data as missing.

I think this is a reasonable summary: basically you can include people in follow-up time up to that point at which their information runs out. So you can view survival analysis as being more efficient, as you can (loosely speaking) include all the available numerator and denominator information (subject to a whole heap of assumptions, well described elsewhere).

To run a valid “non-survival analysis” would require everyone to have the same amount of potential follow-up time (e.g. every cancer patient followed up to at least five years) irrespective of their event status.

---

As a final note on how things can go wrong when not accounting for censoring: As with any research endeavour, there are lots of different ways in which people might naively approach this kind of analysis without accounting for censoring: for example, I've seen some ideas for analysis that would include people in follow-up if they had the event at any time, but only include people with no events in the denominator if they reached the final follow-up time (net result: bad news!)

Answer 4

Update: @James Stanley address the same point in his answer; see point #4.

---

This is not an answer but an extended comment to clarify some terminology and highlight an important assumption.

You write:

if you don't know what happened after Mr. Smith dropped out of your study (i.e. was lost to follow-up, i.e. was censored) ...

implying that "loss to follow-up" is the same as "censoring". Not exactly.

Say we do a randomized control trial (RCT) of treatment T vs control C and the outcome of interest Y is the time to event (or equivalently, survival). We start recruiting subjects on January 1st and the study concludes on December 31, the same year. We follow up with subjects every month from recruitment until Dec 31.

• loss to follow-up: The patient A is recruited and randomized to T or C in Jan. We have data for the first 8 visits and no data for the last 3 visits from Oct to Dec. We know the patient survived for at least 8 months.
• censoring: The patient B is recruited and randomized to T or C in Apr. We have data for all 8 visits until the study concludes in Dec. We know that the patient survived for at least 8 months.

You may ask: Why make the distinction between loss to follow-up and censoring? Patients A and B appear "similar" ("exchangeable"): we followed them for 8 visits post-treatment, so we know they survived at least 8 months, but we don't know how long they survived.

They would contribute the same term $\operatorname{S}(t_i = \text{8 months} | \theta)$ in the likelihood function. [Here I refer to the likelihood $L_t(\theta)$ in @Ben's answer.]

The subtlety is that we don't know why patient A dropped out from the study. Often in analysis when we don't know something, we assume that it happened randomly. This will make the math work out just as described in Ben's answer.

However, suppose that the treatment has severe side effects and patients who received the treatment have a higher probability to discontinue treatment early. This means that subjects in the treament group are more likely to be lost to follow-up. This (potentially) introduces bias in the estimate of the treatment effect, unless more sophisticated analyses come to the rescue.