How trustworthy are biomarker estimates and conclusions in this covid vaccine paper?

Question

My question is about the Correlates of protection against symptomatic and asymptomatic SARS-CoV-2 infection paper from the developers of the ChAdOx1/Vaxzevria vaccine. The paper establishes correlates of vaccine protection: biomarker levels that are predictive of vaccine efficacy (VE) against symptomatic Covid-19. The paper states

[A] new vaccine ... which produces antibody responses that are above the correlate values reported here in at least 50% of participants ... might be expected to have similar efficacy against the clinical endpoints used in our UK trial, and higher efficacy against more severe endpoints.

Question: How supported is this conclusion by the results in the paper?

Details: A case was defined as Covid-19 with at least one of the primary symptoms: fever $\geq 37.8C$; cough; shortness of breath; anosmia; or ageusia. The median biomarker levels were estimated for the following types of VE:

• any symptom (out of 5)
• 3+ symptoms (out of 5)
• symptom(s) including shortness of breath
• symptom(s) excluding shortness of breath

The table below, based on Tables 2 and S4 of the paper, shows the levels with 95% CI vs VE in % for the IgG biomarker (the other biomarkers were similar). A CI bound could not be computed (NC) if it was outside the assay range.

I have the following concerns:

1. The 28 cases with shortness of breath had the usual pattern: higher biomarker levels (=stronger immune response) implied higher efficacy. However, this did not hold for the 24 cases without shortness of breath (the actual levels were not given).
2. No biomarker correlation was established for 16 confirmed Covid-19 cases with non-primary symptoms.
3. The CIs were sometimes not computed; extremely wide, eg 454 (249, 10508); or excluded the estimate, eg 13 (73, 153).
4. For VE of 60%, 70% and 80%, higher biomarkers were required to protect against 3+ symptoms than against 1+. This is counter-intuitive, as 3+ is a subset of 1+ cases. Also, assuming that a case with 3+ symptoms is more severe than with 1 or 2, you'd expect that lower biomarkers would protect against the former. Indeed, efficacy against hospitalisation and death is usually higher than against symptomatic covid. (There could be some explanations, e.g. one symptom is so severe that the patient doesn't feel other symptoms; or 3 mildest symptoms occur together. Still, shouldn't the paper address this?)

How valid are these concerns, and what should be made out of the paper conclusions?

Answer 1

Points 1 and 2 are acknowledged in the paper. The authors found no association between immune responses and mild cases that fall into those categories. From page 203:

No serological measurements in our data were shown to correlate with protection against asymptomatic infection or against symptomatic illness with only mild upper respiratory symptoms.

In general, current data suggest that vaccinations of all types protect against severe Covid while providing little protection against asymptomatic or mild cases.

For point 3, the Methods state:

Correlates and their CIs were not computed for assays in which the relationship between antibody and outcome was nonsignificant. Where CIs were outside the range of values of the assay, these are reported as NC.

That makes sense: if you couldn't have observed that value with the technology you use, there is little point to reporting what it might have been. I see that as a conservative approach to reporting the results.

I only see 1 data point where the CI exclude the point estimate, 50% Vaccine Efficacy (VE) with respect to shortness of breath. According to the methods, CI were calculated with the percentile bootstrap method, which can pose problems when there is bias or skew in estimates of values. That single data point is only reported in the un-refereed supplemental data. The continuous plot in Extended Data Figure 6b, copied below, gives a better idea of what's going on with that particular value.

The estimated relative risk for shortness of breath (green line, with 95% CI shaded) is close to 0.5 (VE of 50%) for all anti-spike IgG levels up to about 150 or so, and with extremely wide CI for IgG values of 13 and below. Furthermore, essentially all observed anti-spike IgG levels (density plot shaded blue) were above the point estimate of 13 for 50% VE. I might have preferred a different estimation of bootstrapped CI, but that particular data-point anomaly doesn't seem to be a serious deficiency when the point estimate for such a low VE is below the range of achieved IgG levels.

Point 4 isn't supported by the data you show. The point estimates for 1+ and 3+ VE values are within each other's CI for each VE at 60% and above (see above for the problems with the 50% VE value), so they can't really be distinguished as you propose. For what it's worth, the point estimate for anti-spike IgG required for 90% VE against any symptom is numerically higher than that for 90% VE against 3+ disease (899 vs. 501), although each is within the CI for the other.

Finally, there's a critical part of the quote from the paper that you omitted, highlighted in bold:

[A] new vaccine that works through similar immune mechanisms and which produces antibody responses that are above the correlate values reported here in at least 50% of participants ... might be expected to have similar efficacy against the clinical endpoints used in our UK trial, and higher efficacy against more severe endpoints.

Without that part of the quote, it would be hard to justify that statement. Different types of vaccines draw on different aspects of the immune system, and it's not yet clear which types of vaccines provide which combinations of VE strength and duration in time. It's not unreasonable, however, to expect that vaccines that use adenoviral vectors, like the one examined here, should have similar associations of IgG levels with clinically assessed VE.