Bias-variance decomposition: term for expected squared forecast error less irreducible error

Question

Hastie et al. "The Elements of Statistical Learning" (2009) consider a data generating process $$ Y = f(X) + \varepsilon $$ with $\mathbb{E}(\varepsilon)=0$ and $\text{Var}(\varepsilon)=\sigma^2_{\varepsilon}$.

They present the following bias-variance decomposition of the expected squared forecast error at point $x_0$ (p. 223, formula 7.9): \begin{aligned} \text{Err}(x_0) &= \mathbb{E}\left( [ y - \hat f(x_0) ]^2 | X = x_0 \right) \\ &= \dots \\ &= \sigma^2_{\varepsilon} + \text{Bias}^2(\hat f(x_0)) + \text{Var}(\hat f(x_0)) \\ &= \text{Irreducible error} + \text{Bias}^2 + \text{Variance} .\\ \end{aligned} In my own work I do not specify $\hat f(\cdot)$ but take an arbitrary forecast $\hat y$ instead (if this is relevant).
Question: I am looking for a term for $$ \text{Bias}^2 + \text{Variance} $$ or, more precisely, $$ \text{Err}(x_0) - \text{Irreducible error}. $$

Answer 1

I propose reducible error. This is also the terminology adopted in paragraph 2.1.1 of Gareth, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning, a book which is basically a simplification of ESL + some very cool R code laboratories (except for the fact that they use attach, but, hey, nobody's perfect). I'll list below the reasons the pros and cons of this terminology.

---

First of all, we must recall that we not only assume $\epsilon$ to have mean 0, but to also be independent of $X$ (see paragraph 2.6.1, formula 2.29 of ESL, 2^nd edition, 12^th printing). Then of course $\epsilon$ cannot be estimated from $X$, no matter which hypothesis class $\mathcal{H}$ (family of models) we choose, and how large a sample we use to learn our hypothesis (estimate our model). This explains why $\sigma^2_{\epsilon}$ is called irreducible error.

By analogy, it seems natural to define the remaining part of the error, $\text{Err}(x_0)-\sigma^2_{\epsilon}$, the reducible error. Now, this terminology may sound somewhat confusing: as a matter of fact, under the assumption we made for the data generating process, we can prove that

$$ f(x)=\mathbb{E}[Y\vert X=x]$$

Thus, the reducible error can be reduced to zero if and only if $\mathbb{E}[Y\vert X=x]\in \mathcal{H}$ (assuming of course we have a consistent estimator). If $\mathbb{E}[Y\vert X=x]\notin \mathcal{H}$, we cannot drive the reducible error to 0, even in the limit of an infinite sample size. However, it's still the only part of our error which can be reduced, if not eliminated, by changing the sample size, introducing regularization (shrinkage) in our estimator, etc. In other words, by choosing another $\hat{f}(x)$ in our family of models.

Basically, reducible is meant not in the sense of zeroable (yuck!), but in the sense of that part of the error which can be reduced, even if not necessarily made arbitrarily small. Also, note that in principle this error can be reduced to 0 by enlarging $\mathcal{H}$ until it includes $\mathbb{E}[Y\vert X=x]$. In contrast, $\sigma^2_{\epsilon}$ cannot be reduced, no matter how large $\mathcal{H}$ is, because $\epsilon\perp X$.

Answer 2

In a system for which all of the physical occurrences have been properly modeled, the left over would be noise. However, there is generally more structure in the error of a model to data than just noise. For example, modelling bias and noise alone do not explain curvilinear residuals, i.e., unmodelled data structure. The totality of unexplained fraction is $1-R^2$, which can consist of misrepresentation of the physics as well as bias and noise of known structure. If by bias we mean only the error in estimating mean $y$, by "irreducible error" we mean noise, and by variance we mean the systemic physical error of the model, then the sum of bias (squared) and systemic physical error is not any special anything, it is merely the error that is not noise. The term (squared) misregistration might be used for this in a specific context, see below. If you want to say error independent of $n$, versus error that is a function of $n$, say that. IMHO, neither error is irreducible, so that the irreducibility property misleads to such an extent that it confuses more than it illuminates.

Why do i not like the term "reducibility"? It smacks of a self-referential tautology as in the Axiom of reducibility. I agree with Russell 1919 that "I do not see any reason to believe that the axiom of reducibility is logically necessary, which is what would be meant by saying that it is true in all possible worlds. The admission of this axiom into a system of logic is therefore a defect ... a dubious assumption."

Below is an example of structured residuals due to incomplete physical modelling. This represents residuals from ordinary least squares fitting of a scaled gamma distribution, i.e., a gamma variate (GV), to blood plasma samples of radioactivity of a renal glomerular filtered radiopharmaceutical [1]. Note that the more data that is discarded ($n=36$ for each time-sample), the better the model becomes so that reducibility deproves with more sample range.

It is notable, that as one drops the first sample at five minutes, the physics improves as it does sequentially as one continues to drop early samples out to 60 min. This shows that although the GV eventually forms a good model for plasma concentration of the drug, something else is going on during early times.

Indeed, if one convolves two gamma distributions, one for early time, circulatory delivery of the drug, and one for organ clearance, this type of error, physical modeling error, can be reduced to less than $1\%$ [2]. Next is an illustration of that convolution.

From that latter example, for a square root of counts versus time graph, the $y$-axis deviations are standardized deviations in sense of Poisson noise error. Such a graph is an image for which errors of fit are image misregistration from distortion or warping. In that context, and only that context, misregistration is bias plus modelling error, and total error is misregistration plus noise error.