Bayesian inference is a method of statistical inference that relies on treating the model parameters as random variables and applying Bayes' theorem to deduce subjective probability statements about the parameters or hypotheses, conditional on the observed dataset.
Overview
Bayesian inference is a method of statistical inference that treats model parameters as if they were random variables in order to rely on probability calculus and produces complete and unified probabilistic statements about these parameters.
This approach starts with choosing a reference or prior probability distribution on the parameters and then applies Bayes' Theorem to deduce probability statements about parameters or hypotheses, conditional on the data, treating the likelihood function as a conditional density of the data given the (random) parameter. Bayes' Theorem asserts that the conditional density of the parameter $\theta$ given the data, $P(\theta|d)$, can be expressed in terms of the density of the data given $\theta$ as
$$P(\theta|d) = \dfrac{P(d|\theta)P(\theta)}{P(d)}.$$
$P(\theta|d)$ is called the posterior probability. $P(d|\theta)$ is often called the likelihood function and denoted $L(\theta|d)$. The distribution of $\theta$ itself, given by $P(\theta)$, is called the prior or the reference measure. It encodes previous or prior beliefs about $\theta$ within a model appropriate for the data. There is necessarily a part of arbitrariness or subjectivity in the choice of that prior, which means that the resulting inference is impacted by this choice (or conditional to it). This also means that two different choices of priors lead to two different posterior distributions, which are not directly comparable.
The marginal distribution of the data, $P(d)$ (which appears as a normalization factor), is also called the evidence, as it is directly used for Bayesian model comparison through the notions of Bayes factors and model posterior probabilities.
The comparison of two models (including two opposed hypotheses about the parameters) in the Bayesian framework indeed proceeds by taking the ratio of the evidences for these two models under comparisons, $$ B_{12} = P_1(d)\big/P_2(d)\,. $$ This is called the Bayes factor and it is usually compared to $1$.
Bayes' formula can be used as an updating procedure: as more data become available, the posterior can be updated successively, becoming the prior for the next step.
References
The following threads contain lists of references:
- What is the best introductory Bayesian statistics textbook?
- Bayesian statistics tutorial
- What is a good book about the philosophy behind Bayesian thinking?
- What is an uninformative prior?
- The meaning of marginals
The following journal is dedicated to research in Bayesian statistics:
- Bayesian Analysis (Open Access)
