What is the asymptotic distribution for the MLE estimator of p(1-p) in a binary random variables

Question

Let $X_i$ be a binary random variable where $P(X_i = 1) = p$. I want to find the MLE estimator for $\theta = p(1-p)$.

The likelihood function should be $$ L(\theta) = \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i} = p^{\sum_{i=1}^nx_i}(1-p)^{n- \sum_{i=1}^nx_i}, $$ How should I proceed? Seems like I can not just take the derivative with respect to $\theta = p(1-p)$.

Edit: You use the plug in estimator, so you find $\hat{p}_{mle}$ first, then $\hat{\theta} = \hat{p}(1-\hat{p})$.

Now suppose I want to find the asymptotic distribution for $\hat{\theta}$. how do I compute the information matrix?

If this is part of a homework assignment, please edit your question to add the self-study tag. — microhaus, May 26 '23 at 18:23
Consider what happens if you choose a $\theta^* \neq \hat{p}(1-\hat{p})$. You are, implicitly, choosing an estimate of $p \neq \hat{p}$, i.e., one that is not the MLE. Therefore $\theta^*$ cannot be the MLE of $\theta$. — jbowman, May 26 '23 at 18:48
@SextusEmpiricus Yes. I have another question, suppose I want to find the asymptotic distribution for $\theta = p(1-p)$, how do I find the information matrix in this case? — Mondayisgood, May 26 '23 at 18:50
$\theta = p(1-p)$ does not fully specify the likelihood function. How would you define the MLE? — Sextus Empiricus, May 26 '23 at 18:56

Sextus Empiricus · Accepted Answer · 2023-05-28T06:37:40.673

1

There is no MLE of $\theta$.

$$L(\theta) = \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i}$$

This expression of the likelihood function is incorrect. The left-hand side expresses the likelihood as a function of $\theta$, but the right hand side is a function of $p$.

You can not fix this either because $\theta$ is a non-injective function of $p$ and can not be inverted. Given $\theta$ we have two possible values of $p$

$$p=\frac{1}{2}\pm\sqrt{\frac{1}{4}-\theta}$$

so you can not compute the probability of the data given $\theta$ unless you have a second parameter.

how do I compute the information matrix?

If you would have a valid MLE then you could start with the information matrix of some parameter and apply a scaling according to the square of the derivative of the transformation.

However, note that some transformed variable can be biased and the Fisher Information alone is not an indication of asymptotic variance. See this example: Why the variance of Maximum Likelihood Estimator(MLE) will be less than Cramer-Rao Lower Bound(CRLB)?

The special case of finding the variance of the distribution of the statistic $\hat\theta = \hat{p}(1-\hat{p}) = \hat{p}-\hat{p}^2$ can be done more directly by computing $$\begin{array}{} Var(\hat\theta) &=& E[(\hat{p}-\hat{p}^2)] - E[\hat{p}-\hat{p}^2]^2 \\ &=& E[\hat{p}^4-2\hat{p}^3+\hat{p}^2] - E[\hat{p}-\hat{p}^2]^2\\ &=& E[\hat{p}^4]-2E[\hat{p}^3]+E[\hat{p}^2] - (E[\hat{p}]-E[\hat{p}^2])^2 \end{array}$$

which can be expressed in terms of the raw moments of $\hat{p}$ (a binomial distributed variable, scaled by $1/n$)

$$\begin{array}{} E[\hat{p}] &=& p \\ E[\hat{p}^2] &=& p^2 + \frac{p(1-p)}{n} \\ E[\hat{p}^3] &=& p^3 + 3 \frac{p^2(1-p)}{n} \\ E[\hat{p}^4] &=& p^4 + 6 \frac{p^3(1-p)}{n} + 3 \frac{p^2(1-p)^2}{n^2} \\ \end{array}$$

and the variance can be written as

$$Var(\hat\theta) = \frac{2p^4-4p^3+2p^2}{n^2} + \frac{-4p^4+8p^3-5p^2+p}{n}$$

where the second term becomes dominant for large $n$ and is the same result as using the Fisher Information matrix

$$p(1-p)/n \cdot \left(\frac{\text{d}\theta}{\text{d}p}\right)^2 = \frac{p(1-p)(1-2p)^2}{n}$$

In the case of $p=0.5$ this would lead to zero variance (or an infinite value in the information matrix). In that case you can still use the Delta method with a second order derivative as demonstrated in this question: Implicit hypothesis testing: mean greater than variance and Delta Method

edited May 28 '23 at 06:37

answered May 26 '23 at 20:52

Sextus Empiricus

77,915

I found this theorem from casella and Berger, page 320. If $\hat{\theta}$ is the MLE of $\theta$, then for any function $\tau(\theta)$, the MLE of $\tau(\theta)$ is $\tau(\hat{\theta})$. – Mondayisgood May 26 '23 at 23:04
@BudLightD.Va That is only true if the function is an injective function, which is not your case. – Sextus Empiricus May 26 '23 at 23:07
1

C&B called this the invariance property of MLE, if the function is injective, then this is pretty obvious. But this property still holds even if the function is not injective. – Mondayisgood May 26 '23 at 23:12
@BudLightD.Va the property only holds if you redefine the likelihood function to be the supremum of the values $\theta$ that map to $\tau$. – Sextus Empiricus May 26 '23 at 23:14
Yes, In fact that's what they did. They called it the induced likelihood function. – Mondayisgood May 26 '23 at 23:15
1

If you do it like that then the second part of my answer still holds. You get the same variance with a scaling factor $\left(\frac{\text{d}\tau}{\text{d}\theta}\right)^2$. You could see this intuitively with the delta method. (that's for the variance, I am not sure whether the Fisher information is still correctly defined when you use this alternative likelihood, the parameter is not complete – Sextus Empiricus May 26 '23 at 23:23
1

An alternative interpretation of the question--and it's the one I favor--is that given a Binomial sample with unknown parameter $p,$ what is the MLE of $p(1-p)$ and what is its asymptotic distribution? That can be determined but it's not trivial. Thus, instead of responding to some faults in the statement of the question, I am suggesting responding to the underlying question itself--which is interesting. – whuber May 27 '23 at 19:30
@whuber This answer evolved out of comments that became too many and too large. In the second part I give a response to the underlying question by referring to the use of the transformation of the Fisher information matrix with a factor $\left(\frac{\text{d}\tau}{\text{d}\theta}\right)^2$. The Fisher information is just the change of the cross entropy of the distribution with itself and if the parameterization is changed, then the rate of change is different based on that factor. – Sextus Empiricus May 27 '23 at 20:46
Possibly one can arrive at the variance by computing the exact sample distribution for $\hat{\theta} = \hat{p}(1-\hat{p})$, but as the question asks for the variance of the asymptotic distribution, this is not really needed and the linear approximation as with the Delta method works. – Sextus Empiricus May 27 '23 at 20:51
I think $p=1/2$ is a special case worth considering. Clearly the Delta method breaks down there. – whuber May 27 '23 at 21:23
@whuber that's a good point. The Delta method can be resolved, but I wonder if and how it works for the Fisher information matrix. – Sextus Empiricus May 28 '23 at 06:55

What is the asymptotic distribution for the MLE estimator of p(1-p) in a binary random variables

1 Answers1