2

I am struggling with my implementation of the expectation maximization (EM) algorithm for a certain model. The sequence of log likelihood values is not increasing, which is contradicting the theory.

The measured outcome variable y is binary and is generated by one of two distributions:

  • p(y=1|z=0) = xi, a fixed probability
  • p(y=1|z=1,x) = 1/(1+exp(theta'*x))

So either there is a fixed probability of obtaining a 1, or the probability depends on the instance's characteristics x, and is modeled with a logistic regression. You can find a more detailed derivation (with the actual LL and the E and M step) in the following document:

https://dl.dropbox.com/u/46839575/MMapproach.pdf

The setting is customer churn prediction, but I guess no background knowledge is needed to check the modeling approach.

I have implemented this in Matlab, and the problem is that the LL does not increase monotonically. I have checked my code several times, and I do not find any mistakes until now, so I am starting to wonder whether my derivation is correct. Maybe someone of you could point to any incorrect assumptions?

In particular, I started to doubt whether the assumption that my latent class membership is modeled with the person's characteristics is correct (in most (Gaussian) mixture models this is a fixed probability, not varying over different persons). Furthermore, the same characteristics are used to model latent class membership and churn probability (although my model also gives non increasing log likelihood values when the set of characteristics in both models is mutually exclusive).

I hope my explanation is clear.

kjetil b halvorsen
  • 77,844
Thomas
  • 51
  • 4
  • It would help if you could write here the completed likelihood... – Xi'an Feb 04 '13 at 21:06
  • 1
    I glanced thru the attached document and could not see obvious mistakes. Are you really checking that the observed likelihood is increasing? LL is the joint log-likelihood... – Xi'an Feb 04 '13 at 21:12
  • Dear Xi'an,

    thank you very much for your comment. It seems I am indeed looking at the complete log-likelihood, not at the incomplete LL. Am I correct that in my case, the incomplete log-likelihood would look as follows:

    log(L_incomplete) = log(L_complete) - log(mu_i)

    with log(L_complete) = equation (7) and mu_i = as defined on page 3 of my document (sorry for referencing the document again, but I don't have enough characters here).

    Also, how to interpret the complete and incomplete LL? Which one is more relevant?

    Thanks so much for your help!

     – Thomas Feb 05 '13 at 06:34
  • To be precise, it should be log(L_incomplete) = log(L_complete) - sum log(mu_i) ; with the sum over all i. Sorry for the basic questions, but I am a bit confused. – Thomas Feb 05 '13 at 06:45
  • I am not sure $\sum_i \mu_i$ is the right term, it should be $\mathbb{E}[\log f(Z|\theta,y,x]$ (see formula (5.11) in our Monte Carlo book). – Xi'an Feb 05 '13 at 07:30
  • 1
    I seem to have a similar problem many years later -- but on an unrelated example. I am working with this well known coin toss example (see: http://www.nature.com/nbt/journal/v26/n8/full/nbt1406.html#close) but I am unable to see the log-likelihood increasing at each iteration as the theory says. Can someone point me to the exact value that I must check to see if its increasing? – Minaj Nov 28 '15 at 23:49

0 Answers0