Why weighted importance sampling is a biased estimator?

Question

By simple math, we can have $$ E_P[f(X)] = \sum_X f(x)p(x) = \sum_X f(x)\frac{p(x)}{q(x)}q(x) = E_Q[f(X)\frac{P(X)}{Q(X)}], $$ which can be approximated by Monte Carlo sampling in two ways.

1. Normal (ordinary) importance sampling: Let $X_1, \cdots, X_n$ be i.i.d. samples from $Q(X)$, and $\mu_1$ the resulted estimator. $$ \mu_1 = \frac{1}{n} \sum^n f(X_i)\frac{P(X_i)}{Q(X_i)} $$ Throughout the rest of this post, we also denote $w(x) = p(x)/q(x)$ as the importance weight. Therefore, the above estimator can also be written as $$ \mu_1 = \frac{1}{n} \sum^n w(X_i)f(X_i) $$ One can prove that $\mu_1$ is an unbiased estimator by showing $$ E_Q[\mu_1] = \frac{1}{n} \sum^n E_Q[f(X_i)\frac{P(X_i)}{Q(X_i)}] = \frac{1}{n} \sum^n E_P[f(X)] = E_P[f(X)] $$

2. Weighted importance sampling: However, sometimes we do not know exactly $P(X)$ up to the normalizing constant, i.e. we know $g(x)$ instead, s.t. $p(x) = g(x)/Z$. This is more useful in the continuous case as $Z$ will be the integral constant, but we here take the discrete case for example. The importance weight is now revised to $w'(x) = g(x)/q(x)$, and consequently so will be the estimator $$ \mu_2 = \frac{\frac{1}{n} \sum^n w'(X_i)f(X_i)}{\frac{1}{n} \sum^n w'(X_i)} $$ One can show this estimator is consistent by showing that $$ \begin{split} \lim_{n \to \infty} \frac{\frac{1}{n} \sum^n w'(X_i)f(X_i)}{\frac{1}{n} \sum^n w'(X_i)} & = \frac{\lim_{n \to \infty} \frac{1}{n} \sum^n w'(X_i)f(X_i)} {\lim_{n \to \infty} \frac{1}{n} \sum^n w'(X_i)} \\ & = \frac{\sum_X g(x)/q(x) f(x) q(x)} {\sum_X g(x)/q(x) q(x)} \\ & = \frac{ZE_P[f(X)]}{Z} = E_P[f(X)] \end{split} $$ And a fun fact is that $\frac{1}{n} \sum^n w'(X_i)$ is actually also an unbiased estimator of the normalizing constant $Z$, i.e. an integral estimate.

Nevertheless, can anyone show that $\mu_2$ is a biased estimator (as it is claimed to be a trade-off between bias and variance), since I cannot find any formal proof and failed to prove it on my own?

---

Some references:

1. The original paper on weighted importance sampling, where I did not find the proof showing the estimator is unbiased.
2. A possibly related answer by Xi'an, but seems he is jumping over a few steps.

Answer 1

Sorry if I was "jumping over a few steps", the argument seems simple enough to me: if $N$ and $D$ are (not necessarily independent) random variables such that $\mathbb E[N]=\nu$ and $\mathbb E[D]=\delta$ then $$\mathbb E[1/D]\ne \frac1\delta\quad\text{and}\quad\mathbb E[N/D]\ne\frac\nu\delta$$ except in highly specialised circumstances. For once thing, the lhs may be undefined as, e.g., when $D$ is a Normal variate (a fact linked with the Cauchy variate having no mean). For another, assuming $D>0$, the inequality is a consequence of Jensen's inequality: since the mapping $\iota: x\mapsto 1/x$ is strictly convex, $\mathbb E[\iota(D)]>\iota(\mathbb E[D])$.

For a mathematical study of the self- (or auto-) normalised importance sampling estimator, see Agapiou et al. (2014).