How to derive the MLE of a Gaussian mixture distribution

Question

In my self-study, I consider a Gaussian mixture distribution:

$$p(x)= p(k=1) N(x|\mu_1,\sigma^2_1) + p(k=0) N(x|\mu_0,\sigma^2_0)$$

where $p(k=1)+p(k=0)=\pi_1+\pi_0=1$. I am now asked to do three things:

1. Write down the likelihood of the observations as a product over $n$ observations

2. Write down the likelihood as a product over the likelihoods for ${K_0}$ and $K_1$, where $K$ is the set of indices for $k=1$ and $k=0$, respectively.

3. Compute the log-likelihood and maximize for $\mu_0$ and $\sigma_0$.

I am not really sure what I am asked to do. I believe the likelihood is given by:

$$p(x|\pi_0, \pi_1, \mu_0, \mu_1, \sigma_0^2, \sigma_1^2) = \prod_{i=1}^n \bigg[ \pi_1 N(x_i|\mu_1,\sigma^2_1) + \pi_0 N(x_i|\mu_0,\sigma^2_0) \bigg]$$

So the log-likelihood is the sum of the logarithm of the sum in the parentheses. This seems correct. However no closed form solution exists of the derivative nor maximizer.

First, I am not sure which of no. 1 or 2 my likelihood expression solves. I think no. 1 but then, second, I am not sure what I am asked to do in no. 2. I suppose the solution to 2 is easier to maximize in 3. Third, it seems there are two different expressions for the likelihood then, but shouldn't there be just one?

Note: I was thinking for no. 2 along the lines of

$$p(x|\pi_0, \pi_1, \mu_0, \mu_1, \sigma_0^2, \sigma_1^2) = \prod_{i=1}^n \bigg[(\pi_1N(x_i|\mu_1,\sigma^2_1))^{k_i} (\pi_0N(x_i|\mu_0,\sigma^2_0))^{1-k_i} \bigg]$$

but got stuck.

Answer 1

This is the proper start but I wonder at the wording of the exercise. I would have asked the following:

1. Write the likelihood of the sample $(x_1,\ldots,x_n)$ when the $X_i$'s are iid from$$p(x)= \mathbb{P}(K=1) N(x|\mu_1,\sigma^2_1) + \mathbb{P}(K=0) N(x|\mu_0,\sigma^2_0)\qquad\qquad(1)$$and conclude at the lack of closed-form expression for the maximum likelihood estimator.
2. Introducing the latent variables $K_i$ associated with the component of each $X_i$, namely$$\mathbb{P}(K_i=1)=\pi_1=1-\mathbb{P}(K=0)$$and$$X_i|K_i=k\sim N(x|\mu_k,\sigma^2_k)$$show that the marginal distribution of $X_i$ is indeed (1).
3. Give the density of the pair $(X_i,K_i)$ and deduce the density of the completed sample $((x_1,k_x),\ldots,(x_n,k_n))$, acting as if the $k_i$'s were also observed. We will call this density the completed likelihood.
4. Derive the maximum likelihood estimator of the parameter $(\pi_0,\mu_0,\mu_1,\sigma_0,\sigma_1)$ based on the completed sample $((x_1,k_x),\ldots,(x_n,k_n))$.

Answer 2

I continued working on this exercise and came up with a solution. I'd be glad about comments.

Let $\theta=[\pi_0,\pi_1,\mu_0,\mu_1,\sigma_0^2,\sigma_1^2]$

1. The likelihood over N observations is given by: $$ P(x|\theta) = \prod_{i=1}^n \bigg[\pi_0 N(x_i|\mu_0,\sigma_0^2)+\pi_1 N(x_i|\mu_1,\sigma_1^2) \bigg]$$

2. The likelihood written as product over sets $K_0$ and $K_1$ is given by

$$ P(x|\theta) = \prod_{i=1}^n \bigg[ (\pi_0 N(x_i|\mu_0,\sigma_0^2))^{1-k_i}(\pi_1 N(x_i|\mu_1,\sigma_1^2))^{k_i} \bigg]$$

3. The log-likelihood is given by

$$ \ln P(x|\theta) = \sum_{i=1}^n \bigg[ (1-k_i) (\ln \pi_0 + \ln N(x_i|\mu_0,\sigma_0^2))+k_i(\ln \pi_1 + \ln N(x_i|\mu_1,\sigma_1^2)) \bigg] $$

Consequently we can find the MLE for $\mu_0$ and $\sigma^2_0$ in a nearly standard way finding:

$$\hat{\mu}_0 = \frac{1}{\sum_{i_1}^n (1-k_i)} \sum_{i_1}^n (1-k_i) x_i$$

$$\hat{\sigma}^2_0 = \frac{1}{\sum_{i_1}^n (1-k_i)} \sum_{i_1}^n (1-k_i)(x_i - \hat{\mu}_0)^2$$