6

In Bayesian inference, the term $P(D|H)$ is sometimes called the likelihood, as in the example below from Olshausen (2004):

$$ P(H|D) = \frac{P(D|H)P(H)}{P(D)} $$

The term $P(D|H)$ is called the likelihood function and it assesses the probability of the observed data arising from the hypothesis...

However, in his introductory paper to the term likelihood, Etz (2018) states the following:

A critical difference between probability and likelihood is in the interpretation of what is fixed and what can vary. In the case of a conditional probability, $P(D|H)$, the hypothesis is fixed and the data are free to vary. Likelihood, however, is the opposite. The likelihood of a hypothesis, $L(H)$, is conditioned on the data, as if they are fixed while the hypothesis can vary. The distinction is subtle, so it is worth repeating: For conditional probability, the hypothesis is treated as a given, and the data are free to vary. For likelihood, the data are treated as a given, and the hypothesis varies.

In other words, Etz (2018) uses $P(D|H)$ as an example of a conditional probability that is 'the opposite' of likelihood. Assuming this is correct, why is $P(D|H)$ still often called the likelihood in Bayesian inference? Is this incorrect?

tdy
  • 471
  • 1
  • 4
  • 10
fujiu
  • 163
  • 1
    just going from the passage you cited, I guess that the number $P(D|H)$ could be called the $\textit{likelihood}$ of the hypothesis of the hypothesis given the data, or the $\textit{probability}$ of the data given the hypothesis – Simon Segert Nov 28 '21 at 20:55
  • 2
    $P(D=d \mid H=h)$ is the probability that the observation $D$ is $d$ given that the parameter $H$ is $h$. But for those following the likelihood principle, including Bayesians, this is taken as proportional to the likelihood (not probability) of $H=h$ given the observation $D=d$. – Henry Nov 28 '21 at 20:57
  • If you had a biased coin and it came up heads $8$ times out of $10$ attempts, this observation is $729$ times more likely when the probability of heads is $p=0.75$ than when $p=0.25$ – Henry Nov 28 '21 at 21:05
  • 1
    Conceived as a function of $H$, at a given $D$, it's a likelihood function. Note that this is along a different axis than the one where it's a density or proportional to a density. The likelihood needn't even be normalizable. – Glen_b Nov 29 '21 at 03:43
  • 1
    The confusion seems to come from the sentence Likelihood, however, is the opposite. They don't mean opposite as in $P(H|D)$, but opposite as in conditional probability $P(D|H)=f(D)$, but likelihood is $P(D|H)=f(D)$. Also in the next sentence conditioned on the data is a bad choice of words, given they don't mean conditional probability, but rather which parameter is fixed. – Sakari Cajanus Nov 30 '21 at 08:47

2 Answers2

8

Note that the likelihood function as defined by Etz is still given by the same conditional probability. I.e.: $$ L(H) = p(D|H) $$ When we write it as $L(H)$ (or $\mathcal{L}(H)$, or similar), we emphasize that we're evaluating $p(D|H)$ as a function of $H$ (with $D$ fixed), rather than of $D$ (with $H$ fixed).

However, that doesn't mean that it's incorrect to say that $p(D|H)$ is the likelihood, in situations where we are indeed evaluating it as a function of $H$, conditioned on some given value of $D$. The typical situation in which this happens is when applying Bayes' rule, as in Olshausen's example, to obtain the posterior. By definition, that scenario has $D$ fixed, and so it satisfies the criteria for calling $p(D|H)$ the likelihood (or likelihood function).

It is only incorrect (in Etz' view, and many others, including myself) to call $p(D|H)$ the likelihood if you're using it in the "forward" direction, i.e. with $D$ not fixed.

One argument for using the word "likelihood" when we do is that the likelihood function does not (in general) integrate to 1. That is, $\int L(H=h)dh=\int p(D|H=h)dh\neq1$. Thus, the outputs of the function are clearly not proper probabilities (or probability densities). Conversely, we do have $\int p(D=x|H)dx=1$ (using $x$ here to prevent confusion with the $d$ from the integral), which illustrates that in that case, we are dealing with probabilities (or probability densities). You can see that Olshausen's usage is in accordance with this principle: his likelihood does not (necessarily) integrate to 1 either.

Ruben van Bergen
  • 7,539
4

This seems to be mostly matters of convention. Let me say mine, and clarify the actual important properties besides the nomenclature. $$ \text{Likelihood} = L(\theta) = P\left(\text{Observed data} \, \middle| \, \theta\right) $$ Note well, that here we consider the observed data. This is neither a distribution in terms of data (since data was fixed to the observed data) or parameters, $\theta$. $$ \text{Sampling distribution} = P\left(\text{data} \, \middle| \, \theta\right) $$ This is a distribution of the data and could be used e.g., to draw pseudo-data. This would usually only be used frequentist inference (which violates the likelihood principle) though may be used in Bayesian inference in: approximate Bayesian computation, simulation based calibration, for computing e.g., prior predictive distributions, and lastly objective Bayesian priors. Broadly speaking though, as we condition only on the observed data in Bayesian analysis, it obeys the likelihood principle and requires only the likelihood.

tdy
  • 471
  • 1
  • 4
  • 10
innisfree
  • 1,480