Why do people use $\mathcal{L}(\theta|x)$ for likelihood instead of $P(x|\theta)$?

Question

According to the Wikipedia article Likelihood function, the likelihood function is defined as:

$$ \mathcal{L}(\theta|x)=P(x|\theta), $$

with parameters $\theta$ and observed data $x$. This equals $p(x|\theta)$ or $p_\theta(x)$ depending on notation and whether $\theta$ is treated as random variable or fixed value.

The notation $\mathcal{L}(\theta|x)$ seems like an unnecessary abstraction to me. Is there any benefit to using $\mathcal{L}(\theta|x)$, or could one equivalently use $P(x|\theta)$? Why was $\mathcal{L}(\theta|x)$ introduced?

Answer 1

Likelihood is a function of $\theta$, given $x$, while $P$ is a function of $x$, given $\theta$.

Roughly like so (excuse the quick effort in MS paint):

In this sketch we have a single $x$ as our observation. Densities (functions of $x$ at some $\theta$) are in black running left to right and the likelihood functions (functions of $\theta$ at some $x$) are in red, running front to back (or rather back to front, since the $\theta$ axis comes 'forward' and somewhat to the left). The red curves are what you get when you 'slice' across the set of black densities, evaluating each at a given $x$. When we have some observation, it will 'pick out' a single red curve at $x=x_\text{obs}$.

• The likelihood function is not a density (or pmf). It is defined in terms of density but it's a different one at every point. It doesn't integrate (/sum) to 1. It needn't even be normalizable.

• Indeed, $\mathcal L$ may be continuous while $P$ is discrete (e.g. likelihood for a binomial parameter) or vice-versa (e.g. likelihood for an Erlang distribution with unit rate parameter but unspecified shape)

Imagine a bivariate function of a single potential observation $x$ (say a Poisson count) and a single parameter (e.g. $\lambda$) -- in this example discrete in $x$ and continuous in $\lambda$ -- then when you slice that bivariate function of $(x,\lambda)$ one way you get $p_\lambda(x)$ (each slice gives a different pmf) and when you slice it the other way you get $\mathcal L_x(\lambda)$ (each a different continuous likelihood function).

(That bivariate function simply expresses the way $x$ and $\lambda$ are related via your model)

Conversely, with a discrete $\theta$ and a continuous $x$ the likelihood is discrete and the density continuous.

As soon as you specify $x$, you identify a particular $\mathcal L$, that we call the likelihood function of that sample. It tells you about $\theta$ for that sample -- in particular what values had more or less likelihood of giving that sample.

Likelihood is a function that tell you about the relative chance that this value of $\theta$ could produce your data (in that ratios of likelihoods can be thought of as ratios of probabilities of being in the interval from $x$ to $x+dx$), when comparing it to other values for $\theta$.

Answer 2

According to the Bayesian theory, $f(\theta \mid x_1,\ldots,x_n) = \frac{f(x_1,\ldots,x_n|\theta) \cdot f(\theta)}{f(x_1,\ldots,x_n)}$ holds, that is $\text{posterior} = \frac{\text{likelihood} \cdot \text{prior}}{\textrm{evidence}}$.

Notice that the maximum likelihood estimate omits the prior beliefs(or defaults it to zero-mean Gaussian and count on it as the L2 regularization or weight decay) and treats the evidence as constant(when calculating the partial derivative with respect to $\theta$).

It tries to maximize the likelihood by adjusting $\theta$ and just treating $f(\theta\mid x_1,\ldots ,x_n)$ equal to $f(x_1,\ldots,x_n\mid \theta)$ which we can easily get(usually the loss) and keep the likelihood as $\mathcal{L}(\theta\mid \mathbf x)$. The true probability $\frac{f(x_1,\ldots,x_n|\theta) \ldots f(\theta)}{f(x_1,\ldots,x_n)}$ can hardly be worked out because the evidence(the denominator), $\int_{\theta} f(x_1, \ldots,x_n, \theta)d\theta$, is intractable.

Hope this helps.

Answer 3

I agree with @Big Agnes. Here is what my professor taught in class: One way is to think of likelihood function $L(\theta | \mathbf{x})$ as a random function which depends on the data. Different data gives us different likelihood functions. So you may say conditioning on data. Given a realization of data, we want to find a $\hat{\theta}$ such that $L(\theta | \mathbf{x})$ is maximized or you can say $\hat{\theta}$ is most consistent with data. This is same to say we maximize "observed probability" $P (\mathbf{x} | \theta)$. We use $P(\mathbf{x} | \theta)$ to do calculation but it is different from $P(\mathbf{X} | \theta)$. Small $\mathbf{x}$ stands for observed values, while $\mathbf{X}$ stands for random variable. If you know $\theta$, then $P(\mathbf{x} | \theta)$ is the probability/ density of observing $\mathbf{x}$.

Answer 4

I think the other answers given by jwyao and Glen_b are quite good. I just wanted to add a very simple example which is too long for a comment.

Consider one observation $X$ from a Bernoulli distribution with probability of success $\theta$. With $\theta$ fixed (known or unknown), the distribution of $X$ is given by $p(X|\theta)$.

$$P(x|\theta) = \theta^x(1-\theta)^{1-x}$$

In other words, we know that $P(X=1) = 1 - P(X=0) = \theta$.

Alternatively, we could look treat the observation as fixed and view this as a function of $\theta$.

$$L(\theta | x) = \theta^x(1-\theta)^{1-x}$$

For example, in a maximum likelihood setting, we seek to find $\theta$ which maximizes the likelihood as a function of $\theta$. For example, if we observe $X = 1$, then the likelihood becomes

$$L(\theta | x) = \begin{cases} \theta, & 0 \leq \theta \leq 1 \\ 0, & \text{else} \end{cases}$$

and we see that the MLE would be $\hat\theta = 1$.

Not sure that I've really added any value to the discussion, but I just wanted to give a simple example of the different ways of viewing the same function.

Answer 5

When we write $f(z|\omega)$ it means a function of $z$ for parameters $\omega$. The density $P(x|\theta)$ is a function of data $x$ for parameters $\theta$, e.g. $\int_X P(x|\theta)dx=1$.

When we define likelihood function $\mathcal{L}(\theta|x)$ it means a function of parameters $\theta$ given the dataset $x$. For instance, $\int_\Theta\mathcal{L}(\theta|x)d\theta\ne 1$.

It's important distinction because the optimization problem is over parameters $\theta$ and not the dataset $x$:$$\min_\Theta \mathcal{L}(\theta|x)$$ In fact, we have only one dataset in this case. Therefore, despite we define likelihood with equality $\mathcal{L}(\theta|x)=P(x|\theta)$, it is important to denote it as a function of parameters $\theta$.

Answer 6

L(θ) = P(y|x;θ) L(θ): likelihood of θ where θ is not a random variable it is a parameter P(y|x;θ): Probability of observing y given data x when parameter θ is used.

Given data in format of x = [x1, x2, x3, ..., xn] => y (n feature input and y as our predefined output in our training data)

So, maximizing L(θ) helps us get θ parameter which will increase chances of observing y when x given. (Maximizing probability of getting result h(θx) = y)