5

I am given the following dataset:

x = (1,2,3,4,5,6,7,8,9,10)  
y = (25,17,20,26,10,19,20,9,15,10)

Each $ y_x $ follows an exponential distribution with parameter $ \lambda_x $, where $ \lambda_x=b_0 + b_1x $.

Suppose that $ (b_0,b_1) $ have a uniform prior [0,1].

Question: Find the posterior density of $b_0$ and $b_1$

The likelihood function is equal to: $$L(b_0,b_1|x,y)=\prod_1^n\lambda_xe^{-\lambda_x}=\prod_1^n(b_0 + b_1x)e^{-(b_0 + b_1x)}$$

The posterior is proportional to:

$$ f(b_0,b_1) \propto {\rm prior}\times {\rm likelihood} \propto (1)\times\prod_1^n (b_0 + b_1x)e^{-(b_0 + b_1x)} \propto \prod_1^n(b_0 + b_1x)e^{-(b_0 + b_1x)} $$

Question: Generate 1,000 samples (using importance sampling), and use these samples to find the posterior mean and variance of $b_0$ and $b_1$.

I have created $n=1,000$ samples for the posterior distribution using importance sampling.

My understanding of importance sampling:

$$ E(f(b_0,b_1)) = \sum f(b_0,b_1) \times p(b_0,b_1) = \sum f(b_0,b_1) \times p(b_0,b_1) \times \frac{g(b_0,b_1)}{g(b_0,b_1)} = \sum \frac{f(b_0,b_1) \times p(b_0,b_1)}{g(b_0,b_1)} \times g(b_0,b_1) $$

Let $ \frac{p(b_0,b_1)}{g(b_0,b_1)} = w(b_0,b_1) $. Then we get:

$$ E(f(b_0,b_1)) = \frac{\sum f(b_0,b_1) \times w(b_0,b_1)}{\sum w(b_0,b_1)} $$.

How do I use these samples to get information about $b_0$ and $b_1$ (i.e., posterior mean and posterior variance)?

If I want to find the mean of $b_0$, I need to find:

$$ E(b_0)=\sum\limits_{b_0} b_0 \times f(b_0 | x,y) $$ where $ f(b_0 | x,y) $ is the marginal posterior of $b_0$. I am stuck at this point... how do I find $ f(b_0 | x,y) $ for my dataset? Do I just need to compute the following quantity?

$$ E(b_0)=\sum\limits_{b_0} b_0 \times f(b_0,b_1| x,y) $$

jon
  • 53
  • I think that by using uniform priors on restricted sets the posterior distribution will not include values for the slope and intercept parameters outside [0,1]. – Michael R. Chernick Dec 13 '16 at 01:03
  • 1
    Please add the [self-study] tag & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. – gung - Reinstate Monica Dec 13 '16 at 01:13
  • @Michael Chernick: Thank you for your response. I agree - I think that the prior will restrict $b_0$ and $b_1$ to the interval [0,1]. – jon Dec 13 '16 at 01:58
  • @gung: Thank you! I've added the self-study tag. This is my first time using stat exchange, so please forgive me! I understand that the mean of the samples will give me the mean of the posterior, while the variance of the samples will give me the variance of the posterior. However, I don't see how I can obtain the mean and variance for $b_0$ or $b_1$. I have previously used Gibbs sampling to estimate parameters in a posterior distribution. In that case I generated samples for the specific parameter, so finding the mean and variance was trivial (i.e. calculate mean and variance of the chains). – jon Dec 13 '16 at 02:06
  • @JuhoKokkala When I used Gibbs sampling in the past, I generated chains of values for the parameters (i.e. $b_0$ and $b_1$) based on their posterior conditional distributions. I could then find the mean and variance of $b_0$ and $b_1$ by computing the mean and the variance of the chains... In this case, I have generated a sample for just the posterior distribution. Computing the mean and the variance of these samples will give me the mean and the variance of the posterior distribution, which is not what I am looking for... I am looking for the mean and the variance of $b_0$ and $b_1$. – jon Dec 13 '16 at 19:29
  • Can you explain the code? Furthermore, if you are not using Metropolis-Hastings (but (self-normalized) importance sampling), why do you have the Metropolis-Hastings tag? – Juho Kokkala Dec 14 '16 at 17:24
  • @JuhoKokkala I updated my question and explained my code. – jon Dec 14 '16 at 17:43
  • 1
    Do you have some textbook or such from which you have read about importance sampling? (Pretty much any explanation of importance sampling should describe how to evaluate (approximations of) expectations with respect to the target distribution, so this question might essentially be "what is importance sampling"). Alternatively, you may be confused about what a posterior distribution is. Could you define what you mean by the functions $f$, $p$, and $g$ in your description of importance sampling? – Juho Kokkala Dec 15 '16 at 18:54
  • Please correct your notations, most formulas in the question are wrong as for instance sums of terms with no index $i$ or $t$. – Xi'an Dec 16 '16 at 08:23
  • Are you confusing simulation average and expectation? $E[b_0]$ is not a sum over $b_0$'s but an integral$$\int_0^1 b_0 \pi(b_0|\mathbf{y},\mathbf{x})\text{d}b_0=\int_0^1 b_0\int_0^1 \pi(b_0,b_1|\mathbf{y},\mathbf{x})\text{d}b_1\text{d}b_0$$ – Xi'an Dec 16 '16 at 08:26
  • @JuhoKokkala Thanks for your response. I got confused between importance sampling and MCMC sampling techniques (such as Metropolis Hastings)! – jon Dec 20 '16 at 01:08

1 Answers1

1

Thanks to Monte Carlo integration, you do not need to evaluate $p(b|o)$ directly but simply to compute your empirical statistics e.g. for the posterior mean for $b$: \begin{eqnarray} E(b|o) &=& \int b \cdot [\int p(b,a|o) \cdot da] \cdot db \\ &\approx& \frac{1}{N}\int \delta(b_i-b) db \\ &=& \frac{1}{N} \sum_i b_i \end{eqnarray} where the $b_i$ are your N generated samples.

beuhbbb
  • 5,043
  • Thanks for your help. Unfortunately, I still don't understand the solution, as I don't see how I can sum up $b_0$ and/or $b_1$. I only have samples for the posterior distribution, which I have generated using importance sampling. If I had MCMC samples for $b_0$ and $b_1$ (i.e. two chains / sets of samples), then I could find the mean and the variance by the method that you described. Perhaps I am missing something obvious... Thanks again. – jon Dec 13 '16 at 23:19
  • @jon "I only have samples for the posterior distribution", that's all you need! – beuhbbb Dec 14 '16 at 08:22
  • Don't I need the marginal posterior of $b_1$ to find the mean of $b_1$? – jon Dec 14 '16 at 14:37
  • 1
    @jon Do the answers at http://stats.stackexchange.com/questions/134027 and http://stats.stackexchange.com/questions/125570/ answer your question? – Juho Kokkala Dec 14 '16 at 15:45
  • @jon. Sorry I do not get it. Can you elaborate about your IS procedure ? – beuhbbb Dec 14 '16 at 16:02
  • @JuhoKokkala Your answer at http://stats.stackexchange.com/questions/125570/marginalising-over-standard-deviation-of-normal-to-get-the-posterior-on-mean helped (I think, anyways). I still don't understand the solution posted by peuhp on this page. I have updated my question with more information as well as some of my code (i.e. for calculating the posterior distribution and for my sampling technique). – jon Dec 14 '16 at 17:19
  • 1
    Where is this code? – Xi'an Dec 16 '16 at 08:24