1

Suppose we have a prior $p(\theta)$ and a likelihood function $L(\theta|x)$, and that the likelihood $L(\theta|x)$ is intractable somehow (difficult or impossible to compute) and we instead replace it with an approximate likelihood $\tilde{L}(\theta|x)$ and use the approximate posterior $\tilde{p}(\theta|x) \propto p(\theta) \tilde{L}(\theta|x) $ to conduct inference (for example, calculate the posterior mean of $\theta$). Does this violate the likelihood principle?

Intuitively it seems to violate the likelihood principle because you're not conducting inference under the likelihood $L(\theta|x)$ anymore (though I'm not sure how to prove this formally). However, in reality don't you 'choose' (a fairly arbitrary) likelihood function anyway (because you don't actually know what the true data generating model is), so what's stopping you from calling the approximate likelihood $\tilde{L}(\theta|x)$ the 'real' or 'original' likelihood?

For example, suppose we know for a fact that we have data $x$ generated from binomial distribution $L(n,p|x)=B(n,p)$, and I choose some (subjective, not dependent on the data) prior $p(n,p)$ on the model parameters. Suppose however that I replace $L(n,p|x)$ with $\tilde{L}(n,p|x)=\mathcal{N}(np,np(1-p))$ (a normal distribution approximate, discretised if necessary if a continuous distribution is a problem) and conduct inference using the posterior $\tilde{p}(n,p|x) \propto p(n,p) \tilde{L}(n,p|x)$. Would I be violating the likelihood principle in this case?

Would I still be violating the likelihood principle in the above example, if I didn't know $B(n,p)$ was the true model?

user356974
  • 11
  • 1
    It does not violate the likelihood principle because under the hood, it does assume all information lies within the likelihood function. It’s just that we can’t compute it for various reasons. – Daeyoung Apr 30 '22 at 12:34
  • 2
    The question is too vague to be answered, because it all depends on the construction of $\tilde L$. For instance, the Approximate Bayesian computation (ABC) approach does violate the likelihood principle since, in its vanilla version, it simulates pseudo-data from the sampling model. And/or uses insufficient statistics. – Xi'an Apr 30 '22 at 12:45
  • I found a related post. – Daeyoung Apr 30 '22 at 13:06
  • I think I'm confused because I know the statistical model we're using is almost always wrong and not the true data generating model. Therefore, doesn't pretty much all models violate the likelihood principle unless you can be sure that your likelihood is the correct model? Unless I've misunderstood and the likelihood principle just say all inferences have to come from your assumed likelihood model, whether it's correct or not? In that case, why does ABC violate the likelihood principle since I can just argue I'm using a different likelihood (that is not intractable to compute)? – user356974 Apr 30 '22 at 14:25
  • @Xi'an I have also added a simple example to the original post which hopefully explains my confusion. – user356974 Apr 30 '22 at 15:20
  • @DaeyoungLim I have also added a simple example to the original post which hopefully explains my confusion. – user356974 Apr 30 '22 at 15:20
  • I do not think there exists a well-defined Likelihood Principle when one is unsure about the model. For instance, what would a Likelihood Principle be in non-parametric settings? – Xi'an Apr 30 '22 at 15:23

1 Answers1

0

The likelihood principle states that two experiments with the same likelihood functions (up to a multiplicative constant) of the same parameter $\theta$, provide the same evidence on the parameter $\theta$.

So approximating the likelihood function (as in your example) does violate the likelihood principle, in the sense that it gives you a (slightly) different inference than that of someone who observes the same likelihood function and uses it without approximations.

Note however that in the broader Bayesian view, probabilities in general are subjective. The likelihood does not describe the "true" data generating model, but rather what you know (or believe) about the data generating model. So in that sense there's no such thing as the "real" likelihood. The likelihood principle is about deriving inference in way that is consistent with your prior knowledge/beliefs.

J. Delaney
  • 5,380
  • Thank you for your answer. Sadly I cannot upvote you as I'm new. If you don't mind, can I ask you to elaborate on 'The likelihood principle is about deriving inference in way that is consistent with your prior knowledge/beliefs'? From what I understand the Bayesian approach does adhere to the likelihood principle (at least, excluding priors that use the data like Jeffreys or empirical Bayes), which does assume the likelihood you've chosen is the true model? – user356974 Apr 30 '22 at 18:35
  • The Bayesian approach adheres to the likelihood principle. But the interpretation of probability in general, and therefore of the likelihood, is subjective. Think for example about flipping a coin - the physical process itself is actually deterministic, the meaning of describing it probabilistically is that we don't know all the physical variables affecting it, so it looks to us (subjectively) as random – J. Delaney Apr 30 '22 at 18:58