I understand that an estimator can be biased and yet consistent, and for me intuitivly in these cases the bias converge to zero as n goes to infinity, however can it be the case that the bias won't converge to zero in such an estimator?
Asked
Active
Viewed 678 times
1
-
That would defy the definition of consistency (bias converges in probability to 0) wouldn't it? – AdamO Mar 04 '19 at 17:34
-
There is an example of what you're looking for on the Wikipedia article for consistent estimator – Artem Mavrin Mar 04 '19 at 17:41
-
According to Ryan answer below, how can we interpret Wikipedia example? – Loai Ghoraba Mar 04 '19 at 17:57
-
The answer you accepted uses the wrong definition of bias. The bias of an estimator $\widehat{\theta}$ of a parameter $\theta$ is $E_\theta[\widehat{\theta}-\theta]$, not just $\widehat{\theta} - \theta$ – Artem Mavrin Mar 04 '19 at 18:10
-
You are right. Then I don't fully understand Wikipedia example, can you elaborate on it? Is epsilon a constant or a function of n? – Loai Ghoraba Mar 04 '19 at 18:26
-
I have added an answer explaining the example – Artem Mavrin Mar 04 '19 at 18:28
1 Answers
2
This answer is adapted from an example on Wikipedia.
Let $\theta$ be a parameter of interest, let $\delta > 0$ be fixed, and let $(\widehat{\theta}_n)_{n=1}^\infty$ be a sequence of estimators of $\theta$ with the following discrete distribution under $\theta$: $$ \begin{aligned} P_\theta(\widehat{\theta}_n = \theta) &= 1 - 1/n, \\ P_\theta(\widehat{\theta}_n = n \delta + \theta) &= 1/n. \end{aligned} $$
Claim. $\widehat{\theta}_n \overset{P_\theta}{\to} \theta$ as $n \to \infty$ (so the sequence $(\widehat{\theta}_n)_{n=1}^\infty$ is consistent), but the bias $E_\theta[\widehat{\theta}_n - \theta]$ does not converge to $0$ as $n \to \infty$.
Proof. To show consistency, we must show that $$ \lim_{n\to\infty} P_\theta(|\widehat{\theta}_n - \theta| > \varepsilon) = 0 $$ for all $\varepsilon > 0$. Thus, let $\varepsilon > 0$ be given. Note that for any $n$ we have $$ \begin{aligned} P_\theta(|\widehat{\theta}_n - \theta| > \varepsilon) &= P_\theta(|\widehat{\theta}_n - \theta| > \varepsilon, \widehat{\theta}_n = \theta) + P_\theta(|\widehat{\theta}_n - \theta| > \varepsilon, \widehat{\theta}_n = n \delta + \theta) \\ &= P_\theta(|\theta - \theta| > \varepsilon, \widehat{\theta}_n = \theta) + P_\theta(|n \delta + \theta - \theta| > \varepsilon, \widehat{\theta}_n = n \delta + \theta) \\ &= P_\theta(0 > \varepsilon, \widehat{\theta}_n = \theta) + P_\theta(n \delta > \varepsilon, \widehat{\theta}_n = n \delta + \theta) \\ &= 0 + P_\theta(n \delta > \varepsilon, \widehat{\theta}_n = n \delta + \theta) \\ &= \begin{cases} 0, & \text{if $n\delta \leq \varepsilon$,} \\ 1/n, & \text{if $n\delta > \varepsilon$.} \end{cases} \end{aligned} $$ In particular, we see that as $n \to \infty$, $P_\theta(|\widehat{\theta}_n - \theta| > \varepsilon) \to 0$, so $(\widehat{\theta}_n)_{n=1}^\infty$ is a consistent sequence of estimators.
However, we have $$ \begin{aligned} E_\theta[\widehat{\theta}_n - \theta] &= (\theta - \theta) P_\theta(\widehat{\theta}_n = \theta) + (n \delta + \theta - \theta) P_\theta(\widehat{\theta}_n = n \delta + \theta) \\ &= n \delta P_\theta(\widehat{\theta}_n = n \delta + \theta) \\ &= \delta \end{aligned} $$ for all $n$, so the bias is constant and positive.
Artem Mavrin
- 3,957
-
Okay all good, but can such sequence with such distribution exists? – Loai Ghoraba Mar 04 '19 at 18:34
-
@LoaiGhoraba formally yes, but this is a pathological example constructed to show what can go wrong, so you wouldn't expect to see it in practice (for instance, it requires $\theta$ to be known...) – Artem Mavrin Mar 04 '19 at 18:36