8

This article contains an example of Bayesian vs. Frequentist philosophies.

An old drug successfully treats 70% of patients. To test a new drug, researchers give it to 100 patients, 83 of whom recover. Based on this evidence, how certain should we be that the new drug is worse than, identical to, or better than the old one?

Bayesian Solution: Under standard assumptions the ‘posterior’ probability the new drug is better than the old one is 0.89, the probability it’s the same is 0.11 (starting from a prior of 0.5), and the probability it’s worse is practically zero.

Frequentist Solution: In our case, we need the fraction of the time we’d see 83 or more, or 57 or less recoveries, which is 0.006. This quantity is called a p-value. An oft-criticised convention is that a p-value less than 0.05 implies evidence against the null hypothesis, and our result of 0.006 certainly qualifies as such.

My Question: How were the Bayesian and the Frequentist solutions arrived at? Please provide the steps. Thanks.

thanks_in_advance
  • 765
  • 4
    What do the words "Gaussian" and "central limit theorem" have to do with this particular problem? Please edit your question to make it clear how your thinking has caused you to use them -- otherwise, please remove them, as it is rather confusing. – Sycorax Nov 18 '15 at 23:17
  • @user777 I've removed the 2 terms. The reason I had used them in the first place is that Central Limit Theorem and Gaussian distribution (i.e. Normal Distribution) are the foundation of p-values. But perhaps I was reaching. Hence the removal. – thanks_in_advance Nov 19 '15 at 17:19
  • 1
    That's only true if the reference distribution is normal. For example, the reference distribution for a $\chi^2$ test of independence is... $\chi^2$. – Sycorax Nov 19 '15 at 17:23
  • @user777 Thanks. Appreciate the clarification. – thanks_in_advance Nov 19 '15 at 17:23

1 Answers1

2

Brendon Brewer (the author) wrote a follow-up post on his blog that details the nitty-gritty for both the posterior probability and p-value calculations.

The p-value calculations are pretty standard, assuming a binomial likelihood. The Bayesian model he uses is defined as

\begin{align*} p & \sim \text{dirac-uniform-mixture} \\ x \, | \, p & \sim \text{binomial(100, p)} \end{align*}

where the Dirac/uniform mixture is defined for $p \in [0, 1]$ by

\begin{equation*} \pi(p) = \frac{1}{2}\delta(p - 0.7) + \frac{1}{2}. \end{equation*}

So given a model, you can calculate posterior probabilities in the standard way. Brendon (sensibly) uses a discrete approximation in his technical details, but it can be illustrative to grind things out analytically. I'll demonstrate how one can get to the 0.89 posterior probability of the new drug being better than the old; the other calculations proceed similarly.

The posterior is defined by

\begin{equation*} \pi(p \, | \, x) = \frac{\pi(x \, | \, p) \pi(p)}{\pi(x)} \end{equation*}

where $\pi(x \, | \, p)$ is the binomial mass function, and the normalizing function $\pi(x)$ for general $n$ is \begin{align*} \pi(x) & = \int_{0}^{1} \pi(x \, | \, p)\pi(p) dp \\ & = \frac{1}{2}{n \choose x}\left(\int_{0}^{1}\delta(p - 0.7)p^x (1 - p)^{n - x}dp + \int_{0}^{1}p^x (1 - p)^{n - x} dp \right) \\ & = \frac{1}{2}{n \choose x}\left(0.7^x 0.3^{n - x} + B(x + 1, n - x + 1)\right). \end{align*}

Here I've used the fact that $\int \delta(t-x)f(t)dt = f(x)$ (for sufficiently nice $f$) and that $\int_{0}^{1}p^{\alpha - 1}(1 - p)^{\beta - 1}dp = B(\alpha, \beta)$ for $B$ the beta function (see here).

Given the posterior, one just needs to work through some integration to find $P(p > 0.7 \, | \, x = 83)$:

\begin{align*} \int_{0.7}^{1} \pi(p \, | \, x = 83) dp & = \frac{1}{\pi(x = 83)} \int_{0.7}^{1} \left(\frac{1}{2}\delta(p - 0.7) + \frac{1}{2}\right)\pi(x = 83 \, | \, p) dp \\ & = \frac{1}{\pi(x = 83)} \int_{0.7}^{1} \frac{1}{2} {100 \choose 83} p^{83} (1 - p)^{17} dp \\ & = \frac{\int_{0.7}^{1} p^{83} (1 - p)^{17} dp}{0.7^{83} 0.3^{17} + B(84, 18)} \\ & = 0.8907679. \end{align*}

jtobin
  • 1,586