How representative is Poisson distribution of the distribution of events in reality?

Question

I've always wondered how good a 'fit' is the Poisson distribution to the events we observe in reality. Almost always I've seen it be used for modeling occurrence of events. (For example, arrival of cars in a parking garage or the number or messages sent/received by computers hosts on a network etc.)

We usually model such events by the Poisson Distribution. Is the distribution just a good first approximation to how things happen in reality? If I observe the number of cars/day or messages/day in the above two examples and those that are output by 'picking from the distribution' how much do they differ? How good an approximation is Poisson? (Is it an approximation?) What is the 'magic' behind Poisson that it just gets it right (intuitively speaking :)?

There are some good starting points if you google derivation of poisson distribution, that show how Poisson is magically derived from binomial distribution where n is large and the chance of an event is small. From there it starts to make sense to use it to model count events. The question I guess is how well do real count events match that smooth extension of the binomial situation. — Peter Ellis, Nov 20 '12 at 03:40

score 5 · Accepted Answer · answered Nov 20 '12 at 09:36

One example I can speak for is supermarket sales of Consumer Packaged Goods (CPG). These are also count events - the supermarket may sell 0 units a day, or 1, or 2 and so on, so the Poisson distribution seems like a good first fit.

However, the underlying binomial distribution @PeterEllis notes does not hold. Yes, we may be able to model the number of customers with a binomial... but some customers will buy 1 unit, some will buy 2 units and some will load their pantries and buy 10 units.

The result will usually be overdispersed, so that a negative binomial distribution fits much better than a Poisson one. (Occasionally, we may even see underdispersion for very fast moving items like milk).

+1. Just thought it was worth mentioning that the Poisson is a special case of the Negative Binomial and that one way of deriving the negative binomial is as a mixture of many different Poisson distributions with different means. — David J. Harris, Nov 20 '12 at 21:15

score 4 · Answer 2 · answered Nov 20 '12 at 21:07

If the things being counted are independent of each other and the rate is constant (or follows a model like in poisson regression) then the Poisson distribution will generally hold quite well. Examples like cars arriving at a garage tend to work fairly well (over periods of time that the rate is fairly constant, including both rush hour and the middle of the night for a garage frequented by 9 to 5 workers would not work well). What time you arrive at the garage will have little or influence on what time I arrive. There are exceptions however in that if 2 people arrange to meet at a given time then they are likely to arrive closer together, if one follows the other then they will be even closer. Also things like a nearby traffic light could cause clumps in the arrivals that would not match a Poisson.

If you want to compare a specific dataset to see if the Poisson is a good match then you can use a hanging rootogram.

Mike Dunlavey · Answer 3 · 2012-11-20T13:54:47.550

As @Stephan says, the straight Poisson may not have enough variance to be a good model of real non-negative integer measurements governed by a hazard function. So, often the negative binomial is used, which has an additional parameter $\alpha > 0$ determining the over-dispersion. I've found it useful to parameterize by $\beta=\ln(\alpha)$ because as the over-dispersion $\alpha$ approaches 0, meaning the negative binomial approaches Poisson, the negative binomial becomes difficult to compute.

Another way to increase the dispersion is zero-inflation, which can be applied to either Poisson or negative binomial. To use that, at each measurement time, first conduct a Bernoulli trial (flip a coin). If the coin is "heads", the measurement is 0. Otherwise the measurement is drawn from the Poisson or negative binomial distribution.

score 2 · Answer 4 · edited Oct 31 '18 at 00:47

I have seen that if the events turn out to be regular then the Poisson model overestimates the variance (logical and obvious), whilst if the events turn out to be clustered then the Poisson model underestimates the variance. The Poisson distribution is generated from a random Poisson point process.

My old textbook recommends Cox, D.R. and Miller, H.D. (1965) The theory of stochastic processes pub. Wiley for further reading. In the introductory book a first-order differential equation is derived for such a random process, which is solved to give the probability of observing no events in time $t$, $P(0,t) = e^{-at}$ where $a$ is the rate of events and $t$ is time, then by considering $P(1,t), P(2,t),$ etc. the general Poisson formula is derived by inspection. C. Chatfield Statistics for technology: a course in applied statistics, 2nd Ed. 1978, pub. Chapman and Hall: see pages 70-75.

Those two examples violate the underlying randomness requirement. If the events are more or less random then the Poisson model is a fair model. Cars arriving a busy town centre car park may be an example of a clustered data set, due to 9 to 5 users, perhaps?

How representative is Poisson distribution of the distribution of events in reality?

4 Answers4