How do I construct the likelihood function for a series of observed Bernoulli-distributed datapoints?

Question

I have observed N (a_i, p_i) pairs each drawn from a different Bernoulli distribution. Here a_i are observed amplitudes and p_i are observed probabilities of success for the i^{th} draw.

I would like to model the full likelihood distribution (and not just via the MLE) with a view to identifying which draws belong to the successful class (and hence the statistics of just these draws, especially any irregular uncertainty distributions etc.).

For example, I have

a_i p_i p_true
10 0.2 0
100 0.9 1
11 0.1 0
99 0.93 1
12 0.25 0

I know the PDFs of the model and of the data (both are obviously Bernoulli distributions), but how do I combine them to obtain residuals that I can use to explore the joint distribution? What distribution does the joint distribution follow and how?

I have tried to unpack the cross-entropy and data and model but don't have a clear solution.

Is the continuous Bernoulli distribution a complete red herring?

Note that ordering doesn't matter in my example. Some other points in response to comments:

The p_i values come from an oracle - these represent the probabilities that the datapoints belong to a common class.
The amplitudes a_i are just weights for the corresponding Bernoulli distributions. The higher the amplitude, the higher the scaling of the Bernoulli distribution in its contribution to the overall process.

See also:

"Weighted" Poisson binomial distribution

Weighted sum of Bernoulli distributions

https://math.stackexchange.com/questions/3481907/sum-of-weighted-independent-bernoulli-rvs

What is the CDF of the sum of weighted Bernoulli random variables?

Please let me know if you need any more information to assist in answering. Thanks as ever!

Is p_i directly observed or is it some modeled value based on a_i? — Sextus Empiricus, Apr 04 '23 at 13:27
What is 'p_true' in your table? Is it the true value of p? How do you know it and why does it only have values 0 and 1? — Sextus Empiricus, Apr 04 '23 at 13:30
"Is the continuous Bernoulli distribution a complete red herring?" This sentence is unclear. What is the 'continuous Bernoulli distribution' and how does it connect with the rest of the text? — Sextus Empiricus, Apr 04 '23 at 13:33
@SextusEmpiricus Good point. I am getting at: Since I'd like to explore (sample from) the full likelihood, do I need to invoke a non-discrete version of the PDF? — jtlz2, Apr 04 '23 at 13:36
"weights for the corresponding Bernoulli distributions." I don't have a clear idea what this means. Weights can occur in all sort of ways. The Bernoulli distribution has only a single parameter $p$, how you include the weight in your problem is unclear. — Sextus Empiricus, Apr 04 '23 at 13:38
Each distribution is scaled in value by the weight (amplitude). — jtlz2, Apr 04 '23 at 13:39
"Each distribution is scaled in value" What do you mean by scaling in value of distributions? I do not get this context. — Sextus Empiricus, Apr 04 '23 at 13:41
Feel free to continue asking Qs - I will add clarification into the question as necessary — jtlz2, Apr 04 '23 at 13:41
I believe that is is better to explain your concrete problem instead of using these abstract terms. None of my questions are really clarified. — Sextus Empiricus, Apr 04 '23 at 13:43

score 0 · Answer 1 · answered Apr 04 '23 at 13:53

I know the PDFs of the model and of the data (both are obviously Bernoulli distributions),

Bernoulli distributions are not characterized by a probability density function (PDF) and instead are characterized by a probability mass function (PMF).

but how do I combine them to obtain residuals that I can use to explore the joint distribution?

In many problems the observations are assumed to be independent and are multiplied. For instance if you have two pmf's depending on some parameter $p$ like $$P(X_1=x_1;p) = f_1(x_1,p)$$ and $$P(X_2=x_2;p) = f_2(x_2,p)$$ then the pmf for the joint probability is $$P(X_1=x_1 \land X_2=x_2 ;p) = f_1(x_1,p)f_2(x_2,p)$$.

For Bernoulli distributions the observations $X_i$ are discrete, but the parameter $p$ is continuous. So if you use this to make a likelihood function, then you get a continuous function.

How do I construct the likelihood function for a series of observed Bernoulli-distributed datapoints?

1 Answers1