1

(Please read until the end)

Consider two ways of writing the exponential distribution:

(A) $\frac{1}{\beta} e^{-\frac{x}{\beta}}$ and

(B) $\theta e^{-x\theta}$

If I estimate $\beta$ or $\theta$ from a random sample $\{x_1,x_2,...,x_n\}$ via the Maximum Likelihood estimation, I get similar expressions:

(A) $\hat{\beta}_{MLE}= \frac{\sum{x_i}}{n}$, and

(B) $\hat{\theta}_{MLE}= \frac{n}{\sum{x_i}}$

The problem occurs when I calculate their bias:

(A) $\mathbb{E}[\hat{\beta}_{MLE}]= \beta $, and hence $\hat{\beta}_{MLE}$ is an unbiased estimator while

(B) $\mathbb{E}[\hat{\theta}_{MLE}] \neq \theta $, and hence $\hat{\theta}_{MLE}$ is not an unbiased estimator. See Bias of the maximum likelihood estimator of an exponential distribution for why it's so.

My question is how can the same parameter be biased and unbiased at the same time?

My question is more at a philosophical level-- Intuitively biasedness should only depend on three things:

  1. Which type of distribution is it?
  2. What kind of estimation algorithm is it?
  3. Which one of the several parameters of the distribution is it?

and NOT on how the distribution function is written.

So effectively, it feels like whatever the bias metric was developed for, seems to not be able to faithfully measure it.

To answer my question can someone put light on an intuitive interpretation of biasedness? What real-life quantity is the bias metric trying to capture? And is there any other metric better than bias that we can substitute it with?

Richard Hardy
  • 67,272
learner
  • 11
  • 4
    It is not the same parameter. Bias is indeed not invariant under re-parametrization, this is a well known property that follows from the fact that $E[f(x)] \ne f(E[x])$ – J. Delaney May 29 '22 at 10:57
  • @J.Delaney Thank you for your comment! I, too, can see why mathematically they're not the same. My question is more at a philosophical level--- given that biasness is a property of the type of the distribution function and the estimation algorithm, why should it depend on the way the distribution is written. – learner May 29 '22 at 19:42
  • 2
    Imagine for example that all apartments in some city are square and their average side length is 10m. Then the average apartment area is not (10m)$^2$. why ? this is just a property of the average - it's not a full representation of the underlying distribution. similarly bias is an averaged quantity (average error of an estimator) so it too depend on the specific choice of parametrization – J. Delaney May 29 '22 at 20:20
  • 1
    Consider this fact that I find amazing: $S^2$, the usual sample variance, is unbiased for variance, but $S$ is biased for standard deviation! – Dave May 29 '22 at 20:58
  • 1
    I don't see why it is intuitive that "biasness" should be independent of how the distribution function is written! After all, as @J.Delaney observes, we know that $\mathbb{E}f(x) \neq f(\mathbb{E}x)$, unless $f$ is an affine transformation, so we know that $\mathbb{E}\widehat{(1/\theta)}) \neq 1/\mathbb{E}\hat{\theta}$. Why would we expect otherwise? – jbowman May 29 '22 at 21:03

1 Answers1

2

Answered in comments, copied below:

It is not the same parameter. Bias is indeed not invariant under re-parametrization, this is a well known property that follows from the fact that $E[f(x)] \ne f(E[x])$ - J. Delaney

One parameter is the mean, the other parameter is the inverse of the mean.

kjetil b halvorsen
  • 77,844