8

I am trying to understand the motivation behind Variational Bayes. I get that the posterior $p(z|x)$ can be intractable, when we would have to compute the evidence with $p(x) = \int p(x|z)p(z) \text{d}z$.

However, all tutorials I have read on Variational Bayes so far simply assume that the likelihood $p(x|z)$ is easy to compute. Why? Why does the same assumption not hold for the posterior?

If we can define $p(x|z)$ analytically with our model, why can't we do the same for $p(z|x)$?

Tutorials that I have read:

Kilian Obermeier
  • 751
  • I'm not an expert in VA, but in general likelihood is a function of data given parameters, i.e. a distribution assumed over data, where the distributions of priors come from the prior distributions, so you have all the stuff needed to evaluate it. Posterior is a distribution of parameters (unobserved!) given data. To obtain the posterior you need likelihood, priors and Bayes theorem. – Tim Jun 15 '18 at 09:26

1 Answers1

5

The likelihood is often tractable because we are free to pick it and we pick something that is easy to work with. If it was not tractable, we could not compute it and could not use the model.

Note that the posterior and likelihood are coupled through Bayes theorem:

$$p(z|x) = \frac{p(x|z) p(z)}{p(x)}.$$

The consequence is that we are free to pick the likelihood and the prior as we see fit, but once we have done so the posterior is also fixed. And for many interesting likelihoods, the posterior is intractable.

Conversely, if you wanted the posterior to be tractable, I guess you could also do so but then this would render the likelihood intractable.

bayerj
  • 13,773
  • "this would render the likelihood intractable." - In this case, would we be able to use the variational approach and approximate the likelihood analytically using the (in our case easy to compute) posterior? Or is there a specific reason for choosing the likelihood to be easy to work with and thus the posterior to be intractable instead of vice versa? – Kilian Obermeier Jun 15 '18 at 09:51