I-Squared: From Calculation to Concept

Question

In meta-analyses, it's common to report the I-Squared statistic as a measure of heterogeneity of results across studies. The definition given by Higgens 2003 as "the percentage of total variation across studies that is due to heterogeneity rather than chance" seems fairly intuitive, but there are some details aren't clear to me.

1. First, the way it is typically presented, the calculation is: $$I^{2} = (Q-df_{Q})/Q$$ where Q is Cochrane's Q, defined as the weighted sum of squares of the difference of study effect sizes and the overall effect size. How do you get from this calculation to the definition given by Higgens? I can see how Q is a measure of overall variability in your set of studies, but I don't understand how subtracting its degrees of freedom and dividing by Q approximates the portion of variance explained by true heterogeneity.

2. How do we know that the variation measured by I-Squared is due to true heterogeneity and not sampling variability? After all, isn't the observed variance in each study's effect size going to be due to true study-specific differences and random error?

Bonus: are there summary statistics that address some of the small-sample power issues with I-Squared? From what I gather, the best solution is just to not apply I-Squared and Q-test without a rigorous understanding of the underlying study characteristics - that is, you still need domain knowledge to understand the sources of heterogeneity.

Thanks in advance for your help!

Answer 1

Though $I^2$ is commonly reported as an absolute measure of heterogeneity, as you describe, Borenstein et al. (2017) caution against this. It is a proportion (i.e., relative), and only in strict circumstances--when sampling variability is mostly constant--does it become an reasonable approximation of absolute heterogeneity.

To your questions:

1. $Q$ is $\chi^2$ distributed, and its expected value under $H_0$ is $df$. Thus, the difference between $Q$ and $df$ is taken as the amount of variability (i.e., true heterogeneity) on top of what is expected if the null is true (i.e., just sampling error). By dividing the difference by $Q$, you get a proportion of variability above the null value over total variability.
2. Because $df$ is what we would expect if it was all sampling error. And if $Q$ = $df$, $I^2$ would = 0. And re: "isn't the observed variance in each study's effect size going to be due to true study-specific differences and random error?", not if you're working under a fixed-effects meta-analysis (i.e., where all observed effects come from the same population, and the only thing separating them is sampling error).

Bonus: if by "small-sample power" you mean low $k$ (i.e., # of studies), then yes, you may run into some problems with the $Q$ test. Namely if you fail to reject $H_0$ for $Q$, then $\tau^2$ (i.e., variance of effect sizes) will be held to 0, and you will also have an $I^2$ of 0. Unfortunately, the other statistical means of evaluating heterogeneity described by Borenstein et al. (2017) are prediction/credibility intervals, but those still require an estimate of $\tau$.

What to do in such a case? If you believe a priori that there is no heterogeneity in your sample of effects, then you should be fine fitting a fixed-effects meta-analysis model, and making conclusions about the average effect in your sample. If, alternatively, you believe a priori that there is heterogeneity in your sample, then you will have to wait until the corpus of effects is large enough to provide sufficient power to reject $H_0$ for $Q$, and estimate $\tau^2$, so that you can make claims about the broader population(s) of effects. But in the mean time, you can settle for interpreting the fixed-effects model, and running with the more limited conclusions it affords.

References

Borenstein, M., Higgins, J., Hedges, L. V., & Rothstein, H. R. (2017). Basics of meta‐analysis: I2 is not an absolute measure of heterogeneity. Research Synthesis Methods, 8(1), 5-18.

Answer 2

To add to @jsakaluk's answer specifically about your second point and your bonus.

It is possible and helpful to give a confidence interval for $I^2$. See this article by Ioannidis and colleagues "Uncertainty in heterogeneity estimates in meta--analyses" available here for arguments in favour of them. Doing this often reveals that the estimate of $\tau^2$ and hence $I^2$ is often very imprecise so inference based on it can be dubious. Note also that as you primary studies become increasingly precisely estimated the value of $I^2$ increases, all else being equal. This has lead Rücker and colleagues to argue in an article entitled "Undue reliance on $I^2$ in assessing heterogeneity may mislead" available here that we should focus on $\tau^2$ instead.

As far as your bonus is concerned I think the last thing we should be doing is looking for an exciting novel summary measure. It is much more important to actually look at the results from the primary studies using one of the many types of plot available in the meta-analysis literature to try to understand it. If possible some form of meta-regression to see where the heterogeneity is coming from is a good idea if we have suitable moderator variables. After all heterogeneity is a fact about the world and we should try to explain it not just say how big it is.