5

I am drafting a teaching session for fellow clinicians to try and provide a somewhat intuitive understanding on how Bayesian statistics differs from the frequentist methods we are taught at medical school. In the session I propose an implausible intervention that is studied and happens to find a difference in sample means with a p-value of 0.048. We then go through comparing the 'oddness' of the results with the 'oddness' of the implausible intervention working. I describe Bayes method of comparing these oddities to try and find which is the least implausible. I turn to this calculator because it uses more concrete inputs than other formulations that need more abstract inputs such as p(data). I then go on to fiddle around with some of the inputs to help build an intuitive sense of what very low p-values achieve.

In my example I use it as follows:

  • Calculate the probability of: H0
  • Based on the probability of: p-value
  • P(H0): 0.999 (I use an average of the class's priors)
  • P(p-value|H0): 0.048 (this is the outcome p-value)
  • P(p-value|¬H0): 0.8 (I have assumed this is the same as study power)

Questions:

  1. Is P(p-value|¬H0) the same as a study power? Or is it only it only the same as the study power if the p-value = the "cut-off p-value". I.e, it varies with the observed p-value?
  2. If not, what is it and how could I estimate it and explain it?
  3. Is my methodology valid?
kjetil b halvorsen
  • 77,844
Harvs
  • 191
  • 2
    "Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence", James O. Berger and Thomas Sellke, Journal of the American Statistical Association, Vol. 82, No. 397 might be relevant here – Christoph Hanck Dec 29 '23 at 14:18
  • 2
    The notation is confusing/incorrect. The p-value should be something like p=Pr{data|H0} where you'd need to define "Pr", "data" and "H0" carefully. (Pr is long-run relative frequency, data is statistic as observed or more extreme, H0 means the null is true and test assumptions hold). Also the power is computed under a specific alternative hypothesis. You can't leave it at "not H0". – dipetkov Dec 29 '23 at 15:15
  • 2
    That being said, it may be interesting to read this blog by Daniel Lakens: The relation between p-values and the probability H0 is true is not weak enough to ban p-values and references therein. Note however that the post ends with "If you really want to make statements about the probability the null-hypothesis is true, given the data, p-values are not the tool of choice (Bayesian statistics is)." – dipetkov Dec 29 '23 at 15:21
  • 4
    Your project seems doomed to fail for the following reasons. Testing a null addresses the question: Are the data reasonably consistent with the null hypothesis? A p-value is a measure of this consistency, and can be regarded as a measure of the absolute plausibility of the null. A different question is: Given the data and two competing hypotheses, what is the relative plausibility of each? Posterior probabilities are Bayesian answers for the relative plausibilty of question. – Graham Bornholt Dec 29 '23 at 20:15
  • 1
    Different questions often require different answers, so it can hardly be surprising that p-values are not useful as measures of relative plausibility (something the Berger and Sellke paper should have acknowledged). P-values work perfectly well for the job that they are designed for. – Graham Bornholt Dec 30 '23 at 07:50
  • 1
    The closest analogue to a p-value in Bayes world is the prior predictive probability. If you squint, you can view the frequentist $H_0$ as a prior with point mass at $\beta_0$. – Durden Dec 30 '23 at 16:10
  • 1
    Thank you all for you input and it seems perhaps that I don't have the necessary depth of understanding of these concepts. Perhaps I should state it in a simpler form. Is the p-value not P(data|H0)? If it is, why can't I use Bayes' Theorem to calculate P(H0|data) provided I have the other necessary inputs? – Harvs Dec 30 '23 at 16:21
  • 1
    "Is the p-value not P(data|H0)?" Not exactly. If x is the observed data then p-value = P(Observing data as extreme or more extreme as x|H0). The reason that frequentists cannot use Bayes Theorem for this application is because when H0 concerns a parameter, it is either true or false so P(H0)) = 0 or 1, making applying Bayes Theorem pointless. In contrast, a Bayesian picks a number (any number) between 0 and 1 for their P(H0), and can then plug that into Bayes Theorem and produce a posterior probability. – Graham Bornholt Dec 30 '23 at 18:38
  • 2
    I think a great session for the clinicians would be to get them to write down their own prior for the example you have, then ask say 5 of them what their priors are, input those priors one at a time into the Bayes calculator to get 5 posteriors, and report the 5 Bayesian answers back to the class. Then have a discussion about the pros and cons of both methods. – Graham Bornholt Dec 30 '23 at 18:46
  • Ok thanks for this. I think I will carry on as planned, but I will be clear to caveat that the posterior generated is not necessarily a true value. But that the process is indicative of how priors and 'p-values'/frequentists statistic combine to produce a posterior that is more suggestive of whether a treatment actually works. I.e that extraordinary claims require extraordinary evidence, and that p<0.05 is generally a weak threshold to base changes in clinical practice on unless the prior is high. I will also describe that there are separate Bayesian methodologies – Harvs Dec 31 '23 at 21:59

1 Answers1

1

Can I convert a frequentist p-value into a Bayesian Posterior Probability

No.

Bayes theorem is

$$P(H_0|\text{data}) = \frac{P(\text{data}|H_0)}{P(\text{data})} \cdot P(H_0)$$

as mentioned in the comments, this $P(\text{data}|H_0)$ is not the same as a p-value.

Related questions

Sextus Empiricus
  • 77,915