0

I'm studying Bayesian inference with the famous Bayes formula. To calculate the posterior, we need prior, likelihood and (sometimes) evidence. When I say "sometimes" it's because there are cases for which we don't need this normalization constant and we make the assumption that the posterior distribution can be approximated by the prior multiplied by the likelihood.

Why do we make this assumption when there are methods (mcmc) for calculating this integral? Is it because there are sometimes conjugate priors and so we already know approximately the posterior distribution without needing to calculate the evidence?

Thank you for your help

Guillaume
  • 21
  • 2
    You always need the normalizing constant to make the posterior a valid probability density. However, the posterior is proportional to the product of prior and likelihood and there are some cases where working with the unnormalized density is fine (like in some MCMC approaches). – Demetri Pananos Mar 14 '24 at 14:05
  • Thank you, so in any case, it's always better to know this normalization constant? For example, in what cases could we work with unnormalized posterior? Taking the beta-binomial distribution as an example? – Guillaume Mar 14 '24 at 14:17
  • If you want to deal with the posterior directly (e.g. compute areas under its surface, or sample from it exactly) then you will need the normalizing constant. Most MCMC techniques do not require the normalizing constant, so if you want to approximately sample from the posterior using MCMC, then you probably won't need it. – Demetri Pananos Mar 14 '24 at 14:44
  • Related question. – Durden Mar 14 '24 at 17:24
  • E.g., you would need the evidence when doing model comparison (Bayes factors), see e.g. https://stats.stackexchange.com/questions/497745/under-what-circumstances-can-an-improper-prior-be-used-in-bayesian-analysis/497799#497799 – Christoph Hanck Mar 14 '24 at 17:27
  • @ChristophHanck Bayes factors are an option, but they're not needed for model comparison (e.g. Vehtari et al 2015). – Galen Mar 14 '24 at 17:29
  • OK, thank you - let me rephrase a little: you would for example need the evidence when you want to compute Bayes factors, for example tp perform model comparison: https://stats.stackexchange.com/questions/497745/under-what-circumstances-can-an-improper-prior-be-used-in-bayesian-analysis/497799#497799 – Christoph Hanck Mar 14 '24 at 17:34

1 Answers1

4

the posterior distribution can be approximated by the prior multiplied by the likelihood

This is incorrect. The posterior density $\pi(\theta|x)$ is exactly determined by the product $\pi(\theta)\ell(\theta|x)$ in the sense that $$\pi(\theta|x)=\dfrac{\pi(\theta)\ell(\theta|x)}{\underbrace{\int\pi(\theta)\ell(\theta|x)\text d\theta}_\text{evidence}}$$

there are methods (mcmc) for calculating this integral

This is also incorrect in that (a) MCMC simulations from the posterior [which indeed do not require the evidence to be known] are rarely helpful to compute the evidence, witness the poor performances of the harmonic mean estimator and (b) simulations only produce an approximation to the integral.

Xi'an
  • 105,342