2

How can I show that normal distribution is a second order approximation to any distribution around the mode?

tei
  • 23
  • 4
  • 1
    The Laplace approximation comes to mind : https://en.wikipedia.org/wiki/Laplace%27s_method – Camille Gontier Dec 10 '20 at 15:49
  • 1
    Depending on what you mean by "second order approximation," the statement is false. If you are referring to density functions, then counterexamples include any distribution that does not have a differentiable density at its mode. If you are referring to CDFs, then counterexamples include all distributions of random variables that have nonzero probability of equaling their mode. – whuber Dec 10 '20 at 18:54

1 Answers1

3

Consider an arbitrary probability distribution $p(\theta)$. We want to approximate the distribution around the mode $\hat{\theta} = argmax \log p(\theta)$. We perform a second order Taylor expansion around $\hat{\theta}$ in log space and we obtain $$ \log p(\theta) = \log p(\hat{\theta}) + \frac{1}{2} (\theta-\hat{\theta})^T \underbrace{\left( \nabla \nabla^T \log p(\hat{\theta})\right)}_{=: \psi} (\theta-\hat{\theta}) + O(\theta^3) $$ Where $\psi$ is the hessian matrix. Note that the first order term disappears as we are at a maximum and hence the first derivative is zero there. This is very similar to a Gaussian pdf in normal space, in fact $$ p(\theta) \approx p(\hat{\theta}) \cdot \exp(- \frac{1}{2}(\theta - \hat{\theta})^T(-\psi) (\theta - \hat{\theta}))$$

Then we can define the Laplace approximation $q$ of $p$ as $$ q(\theta) = \mathcal{N}(\theta, \hat{\theta}, -\psi^{-1})$$ It should be clear that this is proportional to the second order Taylor expansion. May also note that the hessian at the mode is symmetric and negative definite, hence $-\psi$ is symmetric and positive definite.

Manuel Glöckler
  • 116