24

I have been learning about Bayesian statistics, and I often have read in articles

"we adopt a Bayesian approach"

or something similar. I also noticed, less often:

"we adopt a fully Bayesian approach"

(my emphasis). Is there any difference between these approaches in any practical or theoretical sense ? FWIW, I am using the package MCMCglmm in R in case that is relevant.

Joe King
  • 3,805
  • 6
    I don't think that "fully Bayesian" has a rigorous meaning. – Stéphane Laurent Jul 07 '12 at 20:38
  • 4
    @Stephane I am pretty sure that fully Bayesian is the same as Bayesian but the adjective fully is used to emphasize that it is not empirical Bayes. – Michael R. Chernick Jul 07 '12 at 21:38
  • 1
    @Michael this makes sense but I still think the meaning is not universal, and it seems to be confirmed by the several different answers to the question. I would not be surprised that some people say "fully bayesian" to say that they use a subjective prior and not a noninformative one. Another possible situation is when people use the "Bayesian-frequentist predictive distribution" and then move to a purely Bayesian approach. – Stéphane Laurent Jul 08 '12 at 09:01
  • @Stephane i accept your judgement. I think you work in Bayesian statistics more than I do and so have probably heard people use the term in various ways. At least my answer is snesible and partially right. – Michael R. Chernick Jul 08 '12 at 12:33
  • @MichaelChernick yes, your answer is an example of a pseudo-Bayesian approach vs a true Bayesian approach, but there are others such situations – Stéphane Laurent Jul 08 '12 at 12:48

6 Answers6

24

The terminology "fully Bayesian approach" is nothing but a way to indicate that one moves from a "partially" Bayesian approach to a "true" Bayesian approach, depending on the context. Or to distinguish a "pseudo-Bayesian" approach from a "strictly" Bayesian approach.

For example one author writes: "Unlike the majority of other authors interested who typically used an Empirical Bayes approach for RVM, we adopt a fully Bayesian approach" beacuse the empirical Bayes approach is a "pseudo-Bayesian" approach. There are others pseudo-Bayesian approaches, such as the Bayesian-frequentist predictive distribution (a distribution whose quantiles match the bounds of the frequentist prediction intervals).

In this page several R packages for Bayesian inference are presented. The MCMCglmm is presented as a "fully Bayesian approach" because the user has to choose the prior distribution, contrary to the other packages.

Another possible meaning of "fully Bayesian" is when one performs a Bayesian inference derived from the Bayesian decision theory framework, that is, derived from a loss function, because Bayesian decision theory is a solid foundational framework for Bayesian inference.

Stéphane Laurent
  • 19,179
  • Thank you for this. thank you, so the package MCMCglmm being "Fully Bayesian" has nothing to do with using MCMC to derive the estimates and it would it still be fully Bayesian if I have to specify the prior, from which the posterior could be found analytically ? I'm sorry if my question doesn't make sense - I'm still a beginner, but I'm trying to learn ! – Joe King Jul 08 '12 at 12:08
  • 1
    MCMC is just a technique which is useful for simulating the posterior distributions in Bayesian statistics. But it has nothing to do with the Bayesian approach itself. – Stéphane Laurent Jul 08 '12 at 12:46
13

I think the terminology is used to distinguish between the Bayesian approach and the empirical Bayes approach. Full Bayes uses a specified prior whereas empirical Bayes allows the prior to be estimated through use of data.

Michael R. Chernick
  • 42,857
  • Thank you ! I have also seen "empirical Bayes" mentioned here and there, but it never cropped up in things I've read, to the point where I had to think seriously about what it means. I just looked at the wikipedia page which says it is also known as "maximum marginal likelihood", and "an approximation to a fully Bayesian treatment of a hierarchical Bayes model". Hmmm, to be honest I don't understand very much of what is on that page :( – Joe King Jul 07 '12 at 17:22
  • @JoeKing There is a lot of interesting and important uses of empirical Bayes methods. The idea goes back to Herbert Robbins in the 1960s. In the 1970s Efron and Morris showed that the James-Stein estimator of a multivariate normal mean and other similar shrinkage estimators are empirical Bayes. In his new book on Large Scale Inference, Brad Efron shows how empirical Bayes methods can be used for problems sometimes called small n large p because many hypotheses are on parameters are tested with relatively small sample sizes (i.e. p can be a lot larger thaan n). This come up with microarrays. – Michael R. Chernick Jul 07 '12 at 17:40
  • 1
    Thank you again. I have to admit that I don't understand all of what you just wrote but I'm going to use it as my starting point for further study on this matter. – Joe King Jul 07 '12 at 17:56
10

I would use "fully Bayesian" to mean that any nuissance parameters had been marginalised from the analysis, rather than optimised (e.g. MAP estimates). For example a Gaussian process model, with hyper-parameters tuned to maximise the marginal likelihood would be Bayesian, but only partially so, whereas if the hyper-parameters defining the covariance function were integrated out using a hyper-prior, that would be fully Bayesian.

Dikran Marsupial
  • 54,432
  • 9
  • 139
  • 204
  • 4
    This seems to be the slightly more general answer. The more quantities that are marginalised rather than optimised the more 'fully Bayesian' the solution is. Empirical Bayes is a special case. – conjugateprior Jul 07 '12 at 22:16
  • Yes, it is only a slight extension on Michaels' answer; essentially optimisation is fundamentally un-Bayesian. – Dikran Marsupial Jul 09 '12 at 10:34
8

"Bayesian" really means "approximate Bayesian".

"Fully Bayesian" also means "approximate Bayesian", but with less approximation.

Edit: Clarification.

The fully Bayesian approach would be, for a given model and data, to calculate the posterior probability using the Bayes rule $$ p(\theta \mid \text{Data}) \propto p(\text{Data} \mid \theta)p(\theta) \>.$$ Except for very simple models, this has too large computational complexity, and approximations are necessary. More accurate approximations, such as using MCMC with Gibbs sampling for all parameters $\theta$, are sometimes called "Fully Bayesian". Less accurate approximations, such as using point estimation for some parameters, cannot be called "Fully Bayesian". Some approximate inference methods are in-between, like Variational Bayes or Expectation Propagation, and they are sometimes (rarely) also called "Fully Bayesian".

Arek Paterek
  • 127
  • Thank you. I read here that MCMCglmm package I am using is Fully Bayesian. Is that because it is using MCMC together with a prior for parameters ? – Joe King Jul 08 '12 at 07:25
  • @Arek I am really not convinced. So when I use a standard conjugate prior I am "more than fully" Bayesian ? And why do you claim that a point estimate is less "accurate" than posterior simulations ? – Stéphane Laurent Jul 08 '12 at 09:04
  • @JoeKing No: your link says that MCMCglmm is a fully Bayesian approach because you can and you have to choose the prior, contrary to the other approaches which are presented. – Stéphane Laurent Jul 08 '12 at 09:06
  • @JoeKing I don't know about the MCMCglmm package, but I guess so. – Arek Paterek Jul 08 '12 at 09:13
  • 1
    @StéphaneLaurent I do not claim that point estimation is always less accurate. Where are the yesterday's comments to my answer? – Arek Paterek Jul 08 '12 at 09:25
  • @Arek so why did you write "Less accurate approximations, such as using point estimation for some parameters" ? But maybe I misunderstand this sentence because I'm French. Concerning the yesterday's comments, I don't know what happens. – Stéphane Laurent Jul 08 '12 at 09:35
  • @StéphaneLaurent Point estimation sometimes will make no visible difference, and sometimes will decrease the accuracy/precision of the approximation. I think I can say that on average point estimation decreases accuracy. Of course it all depends how we define the accuracy of the approximation of the posterior. We may be interested to have a more accurate approximation for some regions of the posterior, and do not need accuracy for different regions. – Arek Paterek Jul 08 '12 at 09:53
  • 1
    @ArekPaterek Your short answer looked like a joke and so those comments which don't apply to your revised answer does not apply to the revised one. So my guess is that a moderator probably removed them. Still calling fully Bayesian approximate is puzzling. – Michael R. Chernick Jul 08 '12 at 12:47
  • @MichaelChernick (+1) My comment has been deleted, I asked whether this answer was a joke. The Bayesian approach and the numerical techniques used to apply the Bayesian approach are two distinct things. – Stéphane Laurent Jul 08 '12 at 13:04
  • 1
    Maybe my first non-deleted comment was not clear. If Arek's answer were right, then how should we call the situation when it is possible to have the exact posterior distribution (such as a simple conjugate prior situation) ? A "more-than-fully" Bayesian approach ? – Stéphane Laurent Jul 08 '12 at 13:12
  • Arek /cc Stéphane, Michael, sorry for any confusion. I have deleted initial comments after your update, in the hope that this will give more opportunity to discuss your reply. – chl Jul 08 '12 at 21:07
  • The question was about the actual usage of the notion "fully Bayesian", and not about what this notion should mean. I think I answered correctly. From what I observe, whether people call their methods "fully Bayesian" (which are never 100% pure Bayesian) depends most often on whether they use point estimation or not, hence it seems to me that this nomenclature really depends on the used degree of approximation. – Arek Paterek Jul 09 '12 at 09:50
  • Side note: that term "fully Bayesian" is used very rarely. The fact that someone called a method "fully Bayesian" in contrast to methods less Bayesian, this subject is practically non-existent and does not deserve this much discussion. – Arek Paterek Jul 09 '12 at 09:57
4

As a practical example:

I do some Bayesian modeling using splines. A common problem with splines is knot selection. One popular possibility is to use a Reversible Jump Markov Chain Monte Carlo (RJMCMC) scheme where one proposes to add, delete, or move a knot during each iteration. The coefficients for the splines are the Least Square estimates.

Free Knot Splines

In my opinion this makes it only 'partially Bayesian' because for a 'fully Bayesian' approach priors would need to be placed on these coefficients (and new coefficients proposed during each iteration), but then the Least Squares estimates do not work for the RJMCMC scheme, and things become much more difficult.

Glen
  • 7,250
4

I would add a characterization that has not been mentioned so far. A fully Bayesian approach "fully" propagates the uncertainty in all the unknown quantities through the Bayes theorem. On the other hand, Pseudo-Bayes approaches such as empirical Bayes do not propagate all the uncertainties. For example, when estimating posterior predictive quantities, a fully Bayesian approach would make use of the posterior density of the unknown model parameters to obtain the predictive distribution for the target parameter. An EB approach would not account for the uncertainty in all the unknowns - for example, some of the hyper-parameters may be set to particular values, thus underestimating the overall uncertainty.

user67724
  • 333