What is the role, if any, of the Central Limit Theorem in Bayesian Inference?

Question

As someone who started by studying classical statistics where the Central Limit theorem is key to making inferences, and only later now am studying Bayesian statistics, I was late to realize that the Central Limit Theorem has a much smaller role to play in Bayesian statistics. Does the Central Limit Theorem play any role at all in Bayesian inference/statistics?

Later Addition: From Bayesian Data Analysis by Gelman et.al. 3rd edition - on the central limit in the Bayesian context. "This result is often used to justify approximating the posterior distribution with a normal distribution" (page 35). I went through a graduate course in Bayesian Statistics without encountering an example in which the posterior was approximated with a normal distribution. Under what circumstances is it useful to approximate the posterior with a normal distribution?

Answer 1

The Frequentist needs asymptotics because the things they are interested in, like intervals which cover the true value 95% of the time or tests which have a false positive rate of less than 5% when the null hypothesis is true, typically do not exist. If the model is linear and the errors Gaussian, we can get exact confidence intervals, but rarely otherwise. However, we can build intervals which cover the truth asymptotically in very broad classes of models by exploiting a quadratic approximation of the likelihood.

The Bayesian does not have this problem. Given a prior and posterior, the 95% credible interval is a very well defined concept: any interval which contains 95% of the posterior mass. Likewise, Bayes factors can be defined in terms of posterior quantities. Life is easier in the linear/Gaussian case because these quantities will be available in closed form. But even in the general case, we can precisely define these quantities mathematically, and thus use the tools of numerical analysis to compute approximations. Most prominent would be Markov-chain Monte Carlo. The Bayesian, given infinite computing power, can thus get arbitrarily close to "correct" credible intervals/posterior means/etc for any sample size and any model.

[Of course, if the prior is not good, these quantities are utterly meaningless. Even if it is, they do not have any guarantee of relating to anything in the "real world" like a frequentist interval does; they are simply the results of "thinking rationally".]

You also ask about how Bayesians might avail themselves of the CLT. This comes in handy if the Bayesian doesn't have infinite computing power. MCMC is guaranteed to work eventually, but it might take too long on your computer. If the reason the posterior is expensive to evaluate is because you have a lot of data, we can deploy a normal approximation to the posterior. Various ways exist to choose the parameters of the approximating normal; perhaps the most popular is the Laplace Approximation, which uses a quadratic approximation of the posterior near its mode (this might remind you of frequentist asymptotics).

Answer 2

The limit theorem is 'central'

The central limit theorem (CLT) has a central role in all of statistics. That is why it is called central! It is not specific to frequentist or Bayesian statistics.

Note the early (and possibly first ever) use of the term 'central limit theorem' by George Pólya who used it in the article "Über den zentralen Grenzwertsatz der Wahrscheinlichkeitsrechnung und das Momentenproblem"

Das Auftreten der Gaußschen Wahrscheinlichkeitsdichte $e^{-x^2}$ bei wiederholten Versuchen, bei Messungsfehlern, die aus der Zusammensetsung von sehr vielen und sehr kleinen Elementarfehlern resultieren, bei Diffusionafurgängen usw. ist bekanntlich aus einem und demselben Grenzwertsatz zu erklären, der in der Wahrscheinlichkeitsrechnung ein zentralen Rolle spielt.

emphasis is mine.

The principle behind the limit is applied whenever we use a normal distribution

The CLT describes the tendency of sums of variables to approach a normal distribution and that is independent from how you would wish to analyse the variables, whether it is frequentist or Bayesian. Such sums occur anyware.

It is arguable that whenever a normal distribution is used, then it is indirectly an application of the central limit theorem. A normal distribution does not occur as an atomic distribution. There is nothing inherently normal distributed and when a normal distribution 'occurs' then it is always due to some process that sums several smaller variables (e.g like a Galton board where a ball is hitting multiple times a pin before ending up in a bin). And such sums can be approximated by a normal distribution.

The use of the normal distribution can have other motivations. For instance, it is the maximum entropy distribution for a given mean and variance. But in that case, it still indirectly relates to the CLT as we can see a maximum entropy distribution as arrising from many random operations that preserve some parameters (like in the case of the normal distribution, the mean and variance are preserved). When we add up many variables with a given mean and variance, then the resulting distribution is likely gonna be something with a high entropy, ie something close to the normal distribution.

The CLT is such a general principle that the question is like asking "what is the role of 'integration' in Bayesian statistics". Or fill in any other trivial process in place of CLT.

Practical application of CLT

It might be that in practice one observes a tendency for textbooks or statisticians/fields to often apply a particular technique, frequentist or Bayesian, and use relatively more or less often a normal approximation. But, that is in principle not related to those fields.

In practice a particular technique might be preferred. For instance when approximating intervals, then one can use a normal distribution as approximation, but that is not a neccesity. One can also use a Monte Carlo simulation to estimate the distribution or sometimes there is a formula for the exact distribution.

Possibly Bayesian approaches use the normal approximation less often because they are in a situation where they use Monte Carlo simulation/sampling already anyway (to find a solution for large intractable models).

It can be that in particular fields the models are too complex to apply a normal distribution approximation and that those fields also often apply Bayesian techniques. That doesn't make the role of the CLT is smaller for Bayesian techniques. At least not in principle.

There is a large amount of scientists that use nothing much more than simple things like ANOVA, chi-squared tests, ordinary least squares fits, or small variations of it. Those techniques happen to be frequentist and use a normal distribution approximation. Because of that it might seem like frequentist techniques often use the CLT but it doesn't rely on it in principle.

---

Related:

How would a bayesian estimate a mean from a large sample?

Would you say this is a trade off between frequentist and Bayesian stats?

Answer 3

Reproduced verbatim from the Wikipedia page:

In Bayesian inference, the Bernstein-von Mises theorem provides the basis for using Bayesian credible sets for confidence statements in parametric models. It states that under some conditions, a posterior distribution converges in the limit of infinite data to a multivariate normal distribution centered at the maximum likelihood estimator with covariance matrix given ${\displaystyle n^{-1}I(\theta _{0})^{-1}}$, where $\theta _{0}$ is the true population parameter and ${\displaystyle I(\theta _{0})}$ is the Fisher information matrix at the true population parameter value.

The Bernstein-von Mises theorem links Bayesian inference with frequentist inference. It assumes there is some true probabilistic process that generates the observations, as in frequentism, and then studies the quality of Bayesian methods of recovering that process, and making uncertainty statements about that process. In particular, it states that Bayesian credible sets of a certain credibility level $\alpha$ will asymptotically be confidence sets of confidence level $\alpha$, which allows for the interpretation of Bayesian credible sets.

With the reference

van der Vaart, A.W. (1998). "10.2 Bernstein–von Mises Theorem". Asymptotic Statistics. Cambridge University Press. ISBN 0-521-49603-9.