1

I have a long series of entities:

x1, x2, x3, ... xn

for each of which, there is a probability of an event occurring. The probability for each x may be different, but they are all independent, and they are all known:

p1, p2, p3, ... pn

I might provide a point-estimate for how many of said event occurred in that series by simply summing the probabilities p, but is it possible to also calculate a confidence interval or some information about the error in a close-form fashion?

The resistance to using some sampling method is that the series of x many be prohibitively long, and there may be many such series upon which the same estimates need be delivered.

EDIT - Refactoring of the question

For a toy example of the problem, imagine I have a large batch of n widgets, each of which has a known probability of failing within the year. Each widget may have a different known probability of failing, some high, some low (say they belong to different widget classes, those classes well-studied in the past). I'm tasked with estimating the number of widgets that will fail at the end of the year for that batch.

I believe I can provide a point estimate by simply summing the probabilities of each widget in the batch. Stakeholders also request a confidence interval, which may not be possible, but I should be able to provide a sense of error/uncertainty/variance around that point estimate.

My understanding is that I can treat my widgets failing/not-failing as a series of Bernoulli trials, and that likely the "number that fail" can be modelled as a Poisson Binomial Distribution, but from there I get stuck.

Josh
  • 308
  • 1
    This doesn't read like a confidence interval question: it looks more like you want to compute a probability. But the vague senses of "entity" and "event" preclude anything more than a guess concerning what you are trying to ask. Could you perhaps give a small concrete example? – whuber Mar 12 '22 at 03:23
  • @whuber An analogous case -- Imagine I made a billion widgets in a batch, and each has a known probability of breaking at the end of some period. I'd like to estimate the total number from that batch that will break and give some sense of the error in my estimate (ideally a confidence interval). – Josh Mar 12 '22 at 06:00
  • That's a probability. Confidence intervals are guesses based on data, whereas you assert all this information is known--there's no guessing involved. However, the second part of your question appears to contradict this: if all the relevant information is known, why are you even considering sampling? – whuber Mar 12 '22 at 14:07
  • @whuber I'm afraid I'm not following your conclusion. Sure, you could state that the final answer of (say) 800M widgets out of 1B widgets is a probability by virtue of its fractional nature, but that is hiding a lot of the story (and possible variance!). Imagine running monte carlo on all those different probabilities to get the estimate - while it would converge to the same, you could also get a sense of the variance. – Josh Mar 12 '22 at 14:20
  • Something vital is missing from your description, then. I believe that you might have a confidence interval question, but you haven't described such a situation. Confidence intervals are based on samples. Please describe your sample. – whuber Mar 12 '22 at 14:44
  • @whuber Maybe think of it this way: If I told a company 800M of the 1B widgets will break, they might reasonably ask for some sense of error bracketing around that number. The monte carlo example I think shows that the hypothetical "800M" can be conceptualized as a point estimate with some variance around that. – Josh Mar 12 '22 at 15:12
  • You don't need a Monte-Carlo simulation for that. The full distribution itself can be computed, although it takes some work; and there are many possible ways to approximate it. Those solutions are given at https://stats.stackexchange.com/questions/5347. The moments of this distribution, however, are easy to calculate. You could use, say, the standard deviation as a measure of dispersion. I hope that your review of that question will either reveal an acceptable answer or show you how to focus your current question on the additional information you need. – whuber Mar 12 '22 at 17:22
  • 1
    @whuber I've refactored the question as best I know how -- my own ignorance limiting further ability. The question linked is helpful though, and I think the Poisson Binomial Distribution sounds like the right direction, but I've got more digesting to do to get there. – Josh Mar 14 '22 at 22:17
  • You're on the right track: that other question does the very job you want. Which solution(s) you adopt will depend on the specifics of your situation: namely, how many widgets you have an what the distribution of their failure probabilities is. But this is not a "point estimate," nor is it a confidence interval or tolerance interval: it is a prediction interval. In concept it is no different than, say, asking for an interval in which a standard Normal variable has a 95% chance to lie in. Can you give some specifics of your set of $p_i$? – whuber Mar 15 '22 at 13:52
  • That makes sense. What I haven't puzzled out yet though is why model it as a Poisson (or employ the other methods), when it seems there must be something more direct since we can calculate the variance of a Poisson Binomial directly. Presumptively it lies in the behavior or the set of probabilities, but I'm not there yet. Namely as unfortunately for any given "batch of widgets", the distribution of probs can vary wildly. Some batches may be center around 0.5 with low sdev, others may even be skewed sharply to the tails. – Josh Mar 15 '22 at 14:33
  • It is not Poisson: "Poisson Binomial" simply is a name for this situation. It's unclear what you mean by "batches." It would be good to have a better understanding of what it is you are trying to model. I am worried you might be dealing with a stochastic process rather than just a probability distribution. Indeed, how do you obtain (what appears to be) perfect knowledge of a huge number of probabilities? – whuber Mar 15 '22 at 15:24
  • You'd have to take my word for it that the probabilities are indeed known, but the short of it is to imagine each widget belonging to a given "class," and each class has been carefully studied historically to get said probabilities. Batches then have different distributions of widgets of any mix of classes. So far though, treating this as a poisson binomial, grabbing it's variance as the sum of (1 - p_i)p_i, and treating the dist to tend towards normal (since every batch is very large) is giving nice results for prediction intervals against monte carlo sims, but more to be done. – Josh Mar 15 '22 at 17:55
  • The details matter: in some cases the Normal approximation works, but in others it is poor: it depends on how unbalanced the probabilities are in each batch. If there are a small number of probabilities dominating the others, the distribution will usually be skewed. Since you can never know probabilities exactly, and since so many are involved, some assessment of the precision with which they are known is warranted because it could have a substantial effect on any results based on combining a large number of probabilities. – whuber Mar 15 '22 at 18:14

2 Answers2

1

I wanted to sum up what ultimately I've found works well for this case and roll up what's been discussed in some of the comments.

Since I am basically wanting to know how many events occurred out of series of independent Bernoulli trials, we can model this as a Poisson Binomial distribution.

The mean of that distribution (which is my estimate) is the sum of all the probabilities.

The variance of that distribution is also known: the sum of all (1 - pi)pi.

That already gives me the standard deviation (the square root of the variance), so I've gotten a little bit of what the stakeholder wants.

We know that often the Poisson Binomial approaches normal with larger number of parameters (constituent probabilities from the trials), and luckily because of the nature of my problem, I will almost always have a large number of parameters. When that's the case, we can get an interval by simply taking the mean +/- 1.96 * sdev.

That's not an assumption I can just make easily though, so I did a bit of modelling to find out find out what are the typical probabilities for a given widget batch (the example scenario above), and what are the more extreme probability distributions, each at the different likely parameter/probability sizes.

I then calculated the Poisson Binomial mean, sdev, and 95% intervals using the approaches noted and ran monte carlo against them, doing this many times (with different draws from the middle and the extremes) to try and get as much coverage of the possible probability distributions I could run into. I compared the mean, sdev, and interval estimates against those retrieved from monte carlo by their percent error (treating the monte carlo result as the "right" answer).

Long story short, the mean and sdev values are in line with Poisson Binomial quite well. And the confidence intervals, while possibly being up to 5% off in the more extreme cases, are well within our business need for low parameter sizes, and have almost no error for higher parameter sizes (which is most of our cases).

So ultimately I used what I can from the known moments from the Poisson Binomial distribution directly, and then back those into intervals assuming normality (which works well enough for our specific use case).

Josh
  • 308
0

The total number of failures will follow a binomial distribution, assuming the probability of failure is constant, and you can just calculate that binomial probability. You can look up “Binomial distribution” for how to do this; Blitzstein and Hwang’s textbook Probability is a great intro for this.

If n is large, the binomial can be approximated as normal.

If the probability is not constant, the calculation becomes more involved. Instead the total number of failures will need to be calculated, either painfully/tediously by hand, or more likely by Monte Carlo methods.

Note that this calculation gives a tolerance interval, not a confidence interval. A confidence interval is for a parameter, e.g. the probabilities, not the observations.

Closed Limelike Curves
  • 1,065
  • 1
    I note that the probability for each widget can differ, that is there are many unique probabilities going into this. My understanding is that the vanilla Binomial distribution is not appropriate in this case. I'm hoping to not resort to Monte Carlo as my conceptual "batch" can be quite larger, and I can have many batches, making it untenable in practice. – Josh Mar 14 '22 at 22:49
  • If your batch is large then the only way to do this in a computationally feasible way will be to either use Monte Carlo, use approximations, or do something like assume a conjugate prior. – Closed Limelike Curves Mar 14 '22 at 22:56
  • I'm perfectly up for approximations so long as the pitfalls it brings are relatively well understand. I'm naive as to what a conjugate prior would entail. – Josh Mar 14 '22 at 23:07
  • The thread I linked to in a previous comment (to the question) contains various answers, some of which show how to perform exact calculations for very large values of $n.$ They are neither "painful" nor "tedious." Moreover, the Normal approximation rarely holds for this Poisson Binomial form of distribution. Finally, this interval is neither a tolerance interval nor a confidence interval (both of which are estimates based on data): it is a prediction interval for a random variable: namely, the number of failures. – whuber Mar 15 '22 at 13:49
  • @whuber Thanks, you're right about it being a prediction interval.

    That being said, the Wikipedia page on the Poisson-Binomial definitely makes it sound like the calculations involved are difficult, as they require techniques like discrete Fourier transforms.

     – Closed Limelike Curves Mar 19 '22 at 17:20