1

Edit: This question has been closed for being unrelated although I see similar questions posted here with the same objective, yet not with enough detailed answers or not exactly what I am looking for (e.g. this link, or this link). I do not intend that you check my algebra for mistakes. I am simply just showing my work to make it easier for the reader and to show that I put effort into solving my problem, however I could have took a wrong approach.

Question: As an exercise I am trying to marginalize a very simple joint multivariate Gaussian distribution, proceeding by completing the square I get stuck somewhere and I do not get the form I am expecting so I am hoping if anyone can advise how to proceed: $$p(y) = \int p(y,x)dx = \int p(y|x)p(x)dx $$ Where $$ p(y|x) = \frac{1}{(2\pi)^{d/2}}e^{-0.5||y-Ax||^2}$$ $A$ is just a deterministic linear transformation and $$p(x) = \frac{1}{(2\pi)^{m/2}}e^{-0.5||x||^2}$$ So completing the square in the above integral, I get: \begin{align} p(y) &= cte\times \int e^{-0.5(y^Ty -2y^TAx + x^TA^TAx + x^Tx)}dx \\\\ &= cte \times \int e^{-0.5(y^Ty + x^Tx + x^TA^TAx - 2y^TAx)}dx \\\\ &= cte \times e^{-0.5(y^Ty)} \int e^{-0.5(x^T(I_m+A^TA)x - 2y^TAx)}dx \\\\ &= cte \times e^{-0.5(y^Ty)} \int e^{-0.5(x-M^{-1}b)^TM(x-M^{-1}b)}dx e^{0.5(b^TM^{-1}b)} \\\\ &= \frac{1}{\sqrt{(2\pi)^{d}|M|}}e^{-0.5(y^Ty - b^TM^{-1}b)} \end{align}

Where the cte is basically just the normalizing constants before the gaussians and $M = I_m + A^TA$, $b = A^Ty$, and we integrate the new Gaussian density. Continuing from here: \begin{align} p(y) &= \frac{1}{\sqrt{(2\pi)^{d}|M|}}e^{-0.5y^T(I_d - AM^{-1}A^T)y} \end{align}

I think mostly think there is an additional step I am missing because I do not retrieve a Gaussian density which I expect to be of this form: \begin{align} p(y) = \frac{1}{\sqrt{(2\pi)^{d}}|AA^T + I_d|^{0.5}}e^{-0.5y^T(AA^T + I_d)^{-1}y} \end{align} I see this result in a lecture note im following (link) but they only state the method which is completing the square, I'd really appreciate any tips or at least a direction towards a reference that does these kinds of detailed derivations.

Please note that I do not want to ``apply" the direct formula or proof, I am interested more in the exact derivations. Thank you so much in advance!

Maths
  • 31
  • Could you explain what you mean by " I do not retrieve a Gaussian density if we look at the scaling factor outside"? Does that mean you obtained an incorrect scaling factor? Or something else? Regardless, this is really a "please check my algebra question," so despite being related to statistics, it's not on topic here. – whuber Jun 08 '23 at 18:38
  • Im sorry this is not within the topic, I am new to this community. In any case, I expect the scaling coefficient to have a denominator which is the determinant of the covariance matrix, in the exponent the covariance matrix is not the same as the coefficient which is |M|. – Maths Jun 08 '23 at 18:48
  • Your algebra goes awry where you introduce $M,$ because $M$ depends on $y$ -- or rather, the expression is replaces depends on $y.$ – whuber Jun 08 '23 at 19:32
  • Thank you very much for the feedback! Can you kindly point out where $M$ depends on y? I forgot to note that $I_m$ is the identity matrix of dimension small $m$, and $A$ is just a deterministic matrix independent of $y$. In the above expressions where I do replacement, only $b$ depends on $y$. – Maths Jun 08 '23 at 19:43
  • Look at the middle line of your derivation: what happened to the $-2y^\prime A x$ term? If if vanished into $b,$ that doesn't change the problem. You need to make the dependence on $y$ fully explicit in your final expression. – whuber Jun 08 '23 at 20:49
  • Thank you again for your time and feedback. I did indeed include it into $b$, so what I understand is that the final expression that I get of $p(y)$ is not useful because of the technique im using to complete the square? It seemed reasonable, because the density I obtain to integrate over $x$ is actually independent of y and has the form of the gaussian density up to a scale, and that's where the $|M|$ pops out at the final part. – Maths Jun 08 '23 at 21:11
  • I think I managed to find an answer to my problem. My derivations are indeed correct, and there is one more additional step to get close to the final form which is based on the Woodbury matrix formula and up until here, this was also suggested in the answer provided in this link link. However that answer was not sufficient for me, and I still needed to show that the determinant of the matrix outside is equivalent to the one in the exponent, which if you do some simple manipulations you find out they are. – Maths Jun 09 '23 at 17:08
  • Exactly: it's a problem in linear algebra. The statistical way to solve it is to ignore the normalizing constant until the very end of the calculation and compute the determinant. That doesn't require the Woodbury formula, by the way, but any valid method would be fine. There's also a geometrical solution (which is the linear algebra in disguise), as I describe at https://stats.stackexchange.com/a/71303/919 (in 2D, but it generalizes in the obvious ways to higher dimensions). – whuber Jun 09 '23 at 17:45
  • 1
    That's actually a very insightful answer and opens up a new perspective for me! Thank you once again Whuber for the constructive feedback and helpful discussion. – Maths Jun 12 '23 at 17:17

0 Answers0