Least squares estimator for parameter p in binomial distribution

Question

I am trying to find the least square estimator for the parameter $p$ in $Bin(n,p)$ but is it even possible? Isn't it the same as finding it using MLE?

Answer 1

As has been pointed out in comments, least squares and maximum likelihood don't always coincide. However, for a Binomial distribution, they do. We have the least squares problem:

$$\min_{\hat{p}} \sum_{i=1}^N(x_i - \hat{p})^2$$

Taking the derivative with respect to $\hat{p}$ and setting it equal to zero leads to:

$$\begin{eqnarray} 0 &=& 2\sum x_i - 2N\hat{p} \\ \hat{p} &=& {1\over N}\sum x_i \end{eqnarray}$$

and checking the second derivative indicates that this is a minimum. We didn't actually have to do the math, as it is well-known that the sample mean minimizes squared error, but for expository purposes there it is.

The maximum likelihood estimator:

$$\max_{\hat{p}} \hat{p}^{\sum x_i}(1-\hat{p})^{N-\sum x_i}$$

Working with logs is easier:

$$\max_{\hat{p}} \sum x_i\log \hat{p} +(N-\sum x_i)\log (1-\hat{p})$$

Taking the derivative leads to:

$$\begin{eqnarray} 0 &=& \sum x_i / \hat{p} - (N-\sum x_i)/(1-\hat{p})\\ 0 &=& (1-\hat{p})\sum x_i - (N - \sum x_i) \hat{p} \\ 0 &=& \sum x_i - \hat{p}\sum x_i - N\hat{p} + \hat{p}\sum x_i \\ N\hat{p} &=& \sum x_i \\ \hat{p} &=& {1 \over N} \sum x_i \end{eqnarray}$$