Is the likelihood a true function?

Question

Many books and many posts on this site define the likelihood as a function of model parameters. However, does the output associated with every possible model parameter have to be unique? For example, it seems that for some two configurations of the model parameter, the observed data can be equally likely.

So, my question is whether we are playing fast and loose with the word "function" when talking about the likelihood or really the likelihood by definition has to be a function and each input for the model parameter must yield a unique $P(x|\theta)$?

Answer 1

Many books and many posts on this site define the likelihood as a function of model parameters.

If you specify a value for each of the parameters*, you will have at most one value for the likelihood.

• (along with everything else you need to have specified, of course)

However, does the output associated with every possible model parameter have to be unique? For example, it seems that for some two configurations of the model parameter, the observed data can be equally likely.

You're confused -- take some function $f$ -- it's fine for $f(x_1)$ and $f(x_2)$ to be equal. $f(x)=(x-3)^2$ is a function, even though $f(2)=f(4)$.

That's two different arguments having the same function value, not the function having two different values for a given argument.

So, my question is whether we are playing fast and loose with the word "function" when talking about the likelihood

Nope

or really the likelihood by definition has to be a function and each input for the model parameter must yield a unique P(x|θ)?

Yes. But you seem to have gotten a little confused about what that means.

Answer 2

A real-valued function $f$ associates to a vector or real entry $θ\in\Theta$ a real number $f(θ)$, that is, \begin{align*}f:\ \Theta &\longrightarrow \mathbb{R}\\ \theta &\longrightarrow f(\theta)\end{align*}Once the data $(x_1,\ldots,x_n)$ is observed and thus fixed, the likelihood associates with a given value of the parameter $θ$ the real number $$\prod_{i=1}^n p(x_i|\theta)$$ where $p(x|\theta)$ is the density of the random variable $X_i$. (I assume i.i.d.-ness in this answer to keep notations at a minimum complexity.) It is therefore a well-defined function in the mathematical sense.

Answer 3

This is closely related to the concept of identifiability in mathematical statistics, which I think relates to your question. Identifiability deals with the possibility that a poorly-specified model may lack a one-to-one relationship between parameter sets and probability distributions over the data, and this causes problems with regard to inference.

For instance, take the "over-parameterized" ANOVA model,

$$ Y_{ij} = \mu + \alpha_i + \epsilon_{ij} , $$

where $1 \leq i \leq k$, $1 \leq j \leq n$, $\epsilon_{ij} \sim$ normal$(0, \sigma^2)$, and no restrictions are placed on$\{ \alpha_i \}_{i=1}^{k}$. Now suppose we were told by an oracle the exact distribution of $Y_{ij}$ within each group, so that we know both its mean and variance for every $i$. (This is in fact the maximum we could ever hope to learn from the data.) Can we recover the model parameters? We cannot, because there's an infinite number of ways we could specify $\mu, \alpha_1, \ldots , \alpha_k$ so that $\text{E}(Y_{ij}) = \mu + \alpha_i$ for each $i$. This would show up in the likelihood function as well, where different parameters sets would give exactly the same likelihood for all possible configurations of the data. The model is not identifiable, and we can't obtain even consistent estimates for any of the mean parameters. For this reason one usually imposes the identifiability constraint $\sum_{i=1}^{k} \alpha_i = 0$.

So while it's important that the parameters of the model specify the distributions involved, it's also important that we be able to go in the other direction and infer parameters from distributions, else we could never uncover the "true" model.