5

I've come across a strange result using a Hartung-Knapp correction (in the metafor::rma() function), when I have a small number of studies, and very similar effect sizes.

What I typically expect is that the Hartung-Knapp correction (appropriately) increases the width of the confidence intervals. I have found an example where the correction dramatically decreases the CIs, by a factor of around 6.

Here's the result without HKNA:

estimate      se    zval    pval    ci.lb   ci.ub 
  0.9078  0.8401  1.0806  0.2799  -0.7388  2.5544

And with HKNA:

estimate      se    tval    pval   ci.lb   ci.ub 
  0.9078  0.1431  6.3435  0.0240  0.2921  1.5235  *

I've always (at least, since Hartung-Knapp CIs in meta-analysis with two trials) argued for HKNA, but I don't understand this.

library(dplyr)
library(metafor)
d1 <- data.frame(tpos = 1, ttot = 11000,
                 cpos = 0, ctot = 11000)
d2 <- data.frame(tpos = 2, ttot = 6000,
                 cpos = 1, ctot = 6000)
d3 <- data.frame(tpos = 1, ttot = 20000,
                 cpos = 0, ctot = 20000)

d <- dplyr::bind_rows(list(d1, d2, d3))
d <- d %>%
  dplyr::mutate(tneg = ttot - tpos,
                cneg = ctot - cpos)
d_es <- metafor::escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=d)
res_hk <- metafor::rma(yi = yi, vi = vi, data = d_es, test = "knha")
res_no_hk <- metafor::rma(yi = yi, vi = vi, data = d_es)
res_hk
res_no_hk
```
Jeremy Miles
  • 17,812

1 Answers1

2

The K&H method can also lead to a reduction in the SE. This happens when the outcomes are unusually homogeneous. Note that the Q-statistic is way below its degrees of freedom here (the expected value of Q under the null hypothesis of homogeneity is equal to its dfs), which indicates that this is such a case. When this happens, the correction factor that is applied to compute the SEs can be below 1.

There is also an (undocumented) option to use an ad-hoc version of the K&H method where the correction factor is constrained to be $\ge$ 1. You can get this with:

res_ah <- metafor::rma(yi = yi, vi = vi, data = d_es, test = "adhoc")
res_ah

The SE will now be unchanged, but use of the t-distribution to construct the CI will lead to a wider CI.

The K&H method has an approximately nominal Type I error rate and coverage, so using the ad-hoc version is actually overly conservative (i.e., the Type I error rate will be below $\alpha$ and the coverage above the nominal rate). However, one may not care about the performance of the method 'on average' and hence could still choose to use this ad-hoc correction.

The following article also discusses this issue:

Jackson, D., Law, M., Rucker, G., & Schwarzer, G. (2017). The Hartung-Knapp modification for random-effects meta-analysis: A useful refinement but are there any residual concerns? Statistics in Medicine, 36(25), 3923-3934.

Wolfgang
  • 16,997