4

If X is a random variable that has Gaussian prior and Gaussian likelihood. What can be inferred about the posterior?

As posterior is proporional to prior*likelihood which are Gaussians, the posterior should also be Gaussians. But I have struggles deriving it.

Victor Popov
  • 43
  • 3
    Welcome to CV. Since you’re new here, you may want to take our tour, which has information for new users. Please add the [self-study] tag and read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. Please make these changes as just posting your homework & hoping someone will do it for you is grounds for closing. If this is self-study rather than homework, let us know, and... it's still a good idea to show us what you've tried or explain specifically what you don't understand. – jbowman Nov 10 '18 at 16:36
  • @Alex, do you mean to say "As posterior is proportional to prior*likelihood" ? – curious_dan Nov 10 '18 at 18:49
  • Possible duplicate of posterior Gaussian distribution although we can probably reverse the duplicate and the target – Sycorax Nov 19 '18 at 17:04
  • Another promising target is my own answer, https://stats.stackexchange.com/questions/124623/gaussian-is-conjugate-of-gaussian/124625#124625 – Sycorax Nov 19 '18 at 17:32
  • @curious_dan : Bayes' formula says the posterior density is proportional to the product of the prior density and the likelihood function. Do you know what the prior density and the likelihood function are? – Michael Hardy Aug 24 '23 at 18:11

1 Answers1

8

The exponents in the prior density and the likelihood are added to each other \begin{align} & \frac{(\mu-\mu_0)^2}{\tau^2} + \frac{(\overline x - \mu)^2}{\sigma^2/n} \\[8pt] = {} & \frac{(\sigma^2/n)(\mu-\mu_0)^2 + \tau^2(\overline x - \mu)^2}{\sigma^2\tau^2/n} \tag 1 \end{align} Now let's work on the numerator: $$ ((\sigma^2/n)+\tau^2) \left(\mu^2 - 2\left(\mu_0\frac{\sigma^2}n + \overline x \tau^2\right)\mu + \text{“constant''} \right) \tag 2 $$ where “constant” means not depending on $\mu$.

Now complete the square: $$ \left(\mu - \left(\mu_0 \frac{\sigma^2}n + \overline x \tau^2\right)\right)^2 + \text{“constant''} $$ (where this “constant” will differ from the earlier one, but it just becomes part of the normalizing constant).

So the posterior density is $$ \text{constant} \times \exp\Big( \text{negative constant} \times (\mu-\text{something})^2 \Big).$$

In other words, the posterior is Gaussian.

The posterior mean is a weighted average of the prior mean $\mu_0$ and the sample mean $\overline x,$ with weights proportional to the reciprocals of the variances $\tau^2$ (for the prior) and $\sigma^2/n$ (for the sample mean).

Michael Hardy
  • 9,818
  • However, just to note that this works when $\mu$ enters linearly in the quadratic form in the Gaussian density of $[X \mid \mu]$. If we have ${x_i - f(\mu)}$, with $f(\cdot)$ nonlinear, then we don’t get a Gaussian posterior. – Dimitris Rizopoulos Nov 10 '18 at 18:21
  • @Henry : Yes. Fixed. $\qquad$ – Michael Hardy Nov 12 '18 at 03:41
  • @MichaelHardy could you elaborate more on what is exactly the value of these constants, because they should follow the form of a Gaussian distribution. – Mosab Shaheen Aug 24 '23 at 09:54
  • @MosabShaheen : I will edit the answer somewhat before commenting further here. – Michael Hardy Aug 24 '23 at 18:57
  • Now I'm thinking my answer above could use some further editing. The answer I considered writing here in comments (and probably still will write) to Mossab Shaheen's request will still probably be essentially what I first thought of writing here. – Michael Hardy Aug 24 '23 at 19:46
  • @MichaelHardy you can write it anywhere (preferably together with the answer). Just I want to know what these constraints equal. – Mosab Shaheen Aug 26 '23 at 01:31
  • @MosabShaheen : The numerator is $$(\sigma^2/n)(\mu-\mu_0)^2 + \tau^2(\overline x - \mu)^2.$$ By routine algebra we see that that is a quadratic polynomial in $\mu{:}$ $$ ((\sigma^2/n)+\tau^2) \left(\mu^2 - 2\left( \frac{\mu_0 \frac{\sigma^2} n
    • \overline x \tau^2}{\frac{\sigma^2}n + \tau^2} \right)\mu + {} \underbrace{ \frac{ \frac{\sigma^2} n \mu_0^2
    • \tau^2\overline x^2}{\frac{\sigma^2}n + \tau^2} } \right) $$

    The part above the $\underbrace{\text{underbrace}}$ does not depend on $\mu;$ thus it is a ``constant.'' To be continued$,\ldots\qquad\qquad\qquad$

     – Michael Hardy Aug 26 '23 at 14:29
  • @MosabShaheen : Now we need to work on the part in the parentheses: $$ \mu^2 - 2\left( \frac{ \mu_0 \frac{\sigma^2} n
    • \overline x \tau^2}{\frac{\sigma^2} n + \tau^2} \right)\mu + {}

    \underbrace{ \frac{ \frac{\sigma^2} n \mu_0^2

    • \tau^2\overline x^2}{\frac{\sigma^2}n + \tau^2} }

    $$ To be continued$,\ldots\qquad$

     – Michael Hardy Aug 26 '23 at 14:31
  • @MosabShaheen : First, complete the square: $$ \overbrace{ \left( \mu^2 - 2 \left( \frac{ \mu_0 \frac{\sigma^2} n
    • \overline x \tau^2}{\frac{\sigma^2} n + \tau^2} \right) \mu
    • \left( \frac{ \mu_0 \frac{\sigma^2} n
    • \overline x \tau^2}{\frac{\sigma^2} n + \tau^2} \right)^2 \right) }^\text{the complete square}

    {} + \underbrace{ \frac{ \frac{\sigma^2} n \mu_0^2

    • \tau^2\overline x^2}{\frac{\sigma^2}n + \tau^2}
    • \left( \frac{ \mu_0 \frac{\sigma^2} n
    • \overline x \tau^2}{\frac{\sigma^2} n + \tau^2} \right)^2

    }_\text{the ``constant'' not depending on $\mu$} $$ To be continued...

     – Michael Hardy Aug 26 '23 at 14:32
  • @MosabShaheen : Now do the first thing you always do immediately after completing the square: $$ \underbrace{\left( \mu
    • \left( \frac{\mu_0\frac{\sigma^2}n
    • \overline x \tau^2}{\frac{\sigma^2} n + \tau^2} \right) \right)^2

    }_\text{the complete square} {} + \underbrace{ \frac{ \frac{\sigma^2} n \mu_0^2

    • \tau^2\overline x^2}{\frac{\sigma^2}n + \tau^2}
    • \left( \frac{ \mu_0 \frac{\sigma^2} n
    • \overline x \tau^2}{\frac{\sigma^2} n + \tau^2} \right)^2

    }_\text{the ``constant'' not depending on $\mu$} $$ To be continued$,\ldots\qquad$

     – Michael Hardy Aug 26 '23 at 14:33
  • @MosabShaheen : So the product of the prior and the likelihood is proportional (as a function of $\mu$) to $$ \begin{align} & \exp\left( -\frac1{2\cdot\text{variance}} \left( \mu - \left( \frac{ \mu_0\frac{\sigma^2}n
    • \overline x \tau^2}{\frac{\sigma^2} n
    • \tau^2} \right) \right)^2 \right) \{} \

    & \qquad {} \times \exp\left( \begin{array}{l} \text{constant not} \ \text{depending on $\mu$} \end{array} \right) \end{align} $$

     – Michael Hardy Aug 26 '23 at 14:40
  • @MosabShaheen : The posterior expected value is therefore $$ \frac{ \mu_0\frac{\sigma^2}n
    • \overline x \tau^2}{\frac{\sigma^2} n
    • \tau^2} = \frac{\frac{\mu_0}{\tau^2} + \frac{n\overline x}{\sigma^2} }{\frac1{\tau^2} + \frac n{\sigma^2}}. $$ Thus it is a weight average of the prior mean $\mu_0$ and the sample mean $\overline x,$ with weight proportional to the reciprocals of the prior variance $\tau^2$ and the sample variance $\sigma^2/n.$ The weight given to the sample mean $\overline x$ grows as the sample size $n$ grows, as on would expect.
     – Michael Hardy Aug 26 '23 at 14:43
  • @MosabShaheen : As the sample size $n$ grows, the weight $$ \frac{\frac 1{\tau^2}}{\frac1{\tau^2} + \frac n{\sigma^2}}$$ given to the prior mean approaches $0$ and the weight $$ \frac{\frac n{\sigma^2}}{\frac1{\tau^2} + \frac n{\sigma^2}}$$ given to the sample mean approaches $1. \qquad$ – Michael Hardy Aug 26 '23 at 14:49