Explanation for unequal probabilities of numbers drawn in a lottery

Question

Drawing 220 times with replacement from a lottery machine with 12 balls numbered 1 to 12 results in the following distribution:

ball no.     1  2  3  4  5  6  7  8  9 10 11 12
frequency   23 18 21 15 24 17 20 16 21 13 19 13

As you can see, the odd numbers are drawn more frequently than the even numbers and the lower numbers more frequently than the higher numbers.

Is that an artifact of the small sample size, or is there some plausible explanation for this unequal distribution?

---

Notes.

1. These are real numbers drawn from a real machine in a real lottery.
2. The machine that was used is not the one pictured but similar to it, i.e. a spherical cage manually operated (not a glass sphere and not an automaton).
3. I investigated this lottery because in the German national lottery ("6 aus 49"), where 6 numbers are drawn from an urn containing 49 numbers, the numbers haven't been drawn with the same frequency in the 5000 or so drawings since 1955 (and several changes of the lottery machine), with 6 being the most frequent number (drawn 611 times) and 13 the most infrequent (492 times). More detailed and complex (descriptive) statistics of the German national lottery can be found here (in German). I'm interested in the present lottery, because it is by a non-profit organization that I trust and support and was curious if their machinery or process might produce an unintenional bias. I don't suspect fraud.

Answer 1

To determine whether the results seem to indicate some shenanigans were afoot, we can test it!

To begin, we need to specify what our null hypothesis is. I'll take the moment here to stress (as subsequent answers have pointed out in more detail) the importance of forming hypotheses before seeing the data we will use for testing this hypothesis. For example you observed that odd and lower-valued numbers appear more often, but our hypothesis should be formed prior to seeing the data. As such, it would be wrong to look at the data, note that odd numbers appear more often in the data, and then test that hypothesis based on the same data (as @Him points out, it's certainly reasonable to collect new data to test this hypothesis).

Based on your writing, it seems like past data has made you believe that inconsistencies may be occurring, so a natural test would be whether the draws are from a (discrete) uniform distribution (the null hypothesis), or whether the draws are from some other distribution which would thus indicating bias towards certain numbers (the alternative hypothesis).

A simple test of this hypothesis is a chi square test. Under the null hypothesis, each number has an equal chance of being drawn (i.e. for each draw, there is a $1/12$ chance that any given number is drawn -- subsequent answers have also shown how a chi square test can be used with different hypothesis such as odd/even difference). Under our null, we can calculate how many draws each number should have had by multiplying $1/12$ by the total number of draws for each number.

The test essentially measures by how much the observed draws differ from the expected number of draws under the null. Using your observed numbers and performing this test (see code below), we find that the p-value is $p = 0.71$, and thus we fail to reject the null that the draws came from a fair process. Thus, we cannot conclude that the draws came from a biased sampling process.

>obs = c(23,18,21,15,24,17,20,16,21,13,19,13) 
>p = rep(1/length(obs),length(obs)) 
>chisq.test(obs,p=p)
Chi-squared test for given probabilities
data:  obs X-squared = 8, df = 11, p-value = 0.7133

As for the second part of the question, suppose we did conclude the draws came from unequal probabilities (which we did not). There are tons of plausible explanations for why this may be. To list two:

1. A hustler with enough skill could probably easily make the process seem fair when it is not, either by cranking the machine in a consistent way that ensures the numbers that started at the bottom end at the bottom after they are done turning the lever, or by simply following the balls while he cranks it.

2. You say it was from a real lottery, but did you actually observe this process? If not, then it's easy to say these were due to a fair lottery even if they were not.

Answer 2

Following @doubled's (+1) chi-squared test, a remaining question is whether 220 draws from the machine are enough to detect an actual small bias. Maybe the odd numbered balls are a bit heavier, lighter, or less round in such a way that they are slightly more likely to be be drawn. Maybe the true probability distribution is $(6,4,6,4, 6,4,6,4, 6,4,6,4)/60.$ What is the the probability a chi-squared test based on 220 draws would detect this bias in favor of odd numbers?

Based on this distribution, one can simulate $m=100\,000$ sessions of 220 draws each, do the chi-squared test each time, and see what fraction of the $m$ sessions rejects the null hypothesis that draws are fair. This gives a good approximation of the power of the chi-squared test to detect the specified degree of unfairness.

To begin, let's look at one such simulated session--which happens not to detect the unfairness (P-value > 5%). [Using R.]

# one session
set.seed(411) # for reproducibility 
pr = c(6,4,6,4, 6,4,6,4, 6,4,6,4)/60
x = sample(1:12, 220, rep=T, p = pr)
TB = tabulate(x);  TB
[1] 23 11 17 20 20 16 19 19 20 17 21 17
chisq.test(TB)$p.val
[1] 0.898677

Now, by simulating $m$ sessions of $n=220$ draws each, we see that we have just a little less than a 50:50 chance of detecting this level of unfairness. At the end of the simulation run vector pv has $m$ P-values, and mean(pv <= 0.05) gives the proportion of rejections. [The parameter nbins=12 of tabulate forces tabulate to give the correct tally, even if a session lacks some of the higher numbers.]

# 100,000 sessions
set.seed(2021)
pr = c(6,4,6,4, 6,4,6,4, 6,4,6,4)/60
m = 10^5;  pv = numeric(m)
for(i in 1:m) {
 x = sample(1:12, 220, rep=T, p = pr)
 TB = tabulate(x, nbins=12)
 pv[i] = chisq.test(TB)$p.val
 }
mean(pv <= 0.05)
[1] 0.45349

Also, a run of the program with $n = 500$ draws per session [not shown] gives power almost 90%, and a run with $n=650$ gives power just above 95%.

Note: In these simple cases, it is not necessary to do a simulation in order to approximate the power of a chi-squared test of $H_0: \mathrm{Fair}$ against alternative vector pr, using $n$ draws.

The 5% critical value $c=19.6751$ has $P(Q > c|H_0) = 0.05.$ And the 'effect size' is $\lambda = n\sum\frac{(p_{ai} - 1/12)^2}{1/12} = 8.8.$ Then the exact power $0.4602$ is found using the chi-squared distribution with degrees $\nu = 12-1 = 11$ and noncentrality parameter $\lambda.$

c = qchisq(.95, 11);  c
[1] 19.67514
lam = 220*sum((pr-1/12)^2/(1/12)); lam
[1] 8.8
1 - pchisq(c,11,lam)
[1] 0.4602406

By contrast with $n = 650,$ we have $\lambda = 26$ and power $0.9574.$

lam = 650*sum((pr-1/12)^2/(1/12)); lam
[1] 26
1 - pchisq(c,11,lam)
[1] 0.9573635

Perhaps see this Q&A and its references.

Answer 3

You mentioned that odd-even pattern, so let's investigate that.

Category	Observed	Expected #	Expected
odd	92	110	50%
even	128	110	50%

And test with just these two categories....

Chi squared equals 5.891 with 1 degrees of freedom.
The two-tailed P value equals 0.0152

That would generally be considered significant This kind of result only happens fifteen times out of a thousand. Does that mean that we have a publishable result?

The binomial test (more exact when there are two categories) gives p = 0.0062, also significant.

There are several problems here. Firstly we chose to focus on one aspect of the data after we had collected it. We ignored others. For example, we could have tested for multiples of 3, or primes/non primes or ... many other choices are possible. Nearly all those other choices find "not significant". This is p-hacking the green jellybean fallacy, illustrated by xkcd. By subdividing or grouping data it is often possible to find a result that is significant. The proper response is to investigate further. If this is a real effect, it should become a stronger result with more data. If it just "green jellybeans" then it won't be repeatable.

And (although not the same experiment) the German 6-aus-49 lottery has no odd-even pattern.

Secondly we have a strong "prior": We inspect the machine and the balls and don't notice anything that seems amiss. We know from our general experience of such machines that they are usually quite fair. What would you estimate the probablity of the machine, or the operator, being substantially unfair, prior to the experiment? Clearly there is some judgement here, but suppose we say 0.001 (one in a thousand chance) Now, given this result, how likely is it that the machine is substantially unfair. Well, this will take you into Bayesian statistics. We would need to quantify "substantially unfair", but if an unfair machine will always give results like this [P(X²>5.891 | unfair)=1] then the probablity that the machine is unfair, given that we observe X²>5.891 is just 0.001/0.0152 =0.06. So we update our estimate of the machine being unfair from 0.001 to 0.06. We still think it rather unlikely that the machine is unfair.

Answer 4

As an addition to the other answers, let me offer you a visual way to inspect the differences between the expected and observed frequencies: A (hanging) rootogram, invented by John Tukey (see also Kleiber & Zeileis (2016)). In the figure below, the square roots of the expected counts are displayed as red dots. The square roots of the observed frequencies are hanging as histogram-like bars from these points. Therefore, if the expected and the observed frequencies are similar, the base of the bars all lie near zero. Bars for lower than expected frequencies lie above the zero line and vice versa. This plot makes it easy to see departures in either direction.

In terms of interpretation, recall that for most values, the square root of a count will be less than one unit from its expected value. This is the case for all twelve numbers of your data. Hence, there is little evidence for a systematic departure from uniformity, as the other answers have already explained. The square root is useful here because it's a variance stabilizing function for the Poisson distribution. With $n=220, p=1/12$, the Poisson distribution is a reasonable approximation to the binomial. Finally, for fairly large $\lambda$s, the variance of the square root of a Poisson distribution will be around $1/4$ so the 68-95-99.7 rule suggests most values will lie within $2\times \sqrt{1/4} = 1$ of the mean. One could entertain the arcsine transformation for the binomial, but it virtually makes no difference in this case.

On a separate note, please keep in mind that the usage of a formal hypothesis test could be dubious in such cases, if you only looked at the data because they made you suspicious (this is called HARKing: Hypothesizing after the results are known). In your case, you formed the hypothesis based on other data, which seems fine to me.

Answer 5

I found that there was a correlation of -0.5109730443013045 between the number drawn and its frequency, and a p-value of 0.090274 for that correlation. For a binomial test for a number being odd or even, I got a p-value of 0.00621804354. When I adjusted the odd numbers for the difference in means (that is, added (even mean-odd mean) to the frequencies of odd numbers), I found a correlation of -0.753098254512706 for the numbers versus frequencies, for a p-value of 0.0047.

As other answers have discussed, this is HARKing (hypothesizing after the results are known). There are lots of different patterns you could have noticed, so given any alpha, the probability that at least one has a p-value less than alpha is significantly greater than the probability for each one individually. Besides the possibility of cherry-picking the hypothesis to fit the data, there is the possibility of cherry-picking the data: if you see lots of lotteries, and only discuss the ones with unusual results, or you stop collecting data from a lottery once that data seems anomalous, and don't continue collecting data to see whether it reverts to the mean, the apparent p-value can misrepresent the true improbability of the results.

0.090274 is a large p-value in general, and in the context of HARKing, it's well within the range of what we could expect by chance. It's enough to draw attention to it, but not enough to come to a solid conclusion. 0.00621804354, on the other hand, is quite small. With it being less that one-eighth of the standard alpha of 0.05, a relevant question to ask is "Are there eight other hypotheses that would be as, or more, noticeable?" It's on the borderline (and keep in mind that you need to account for the bias that the pattern that you actually say feels more salient than ones that you didn't see). When you add in the fact that there are surely more than 161 lotteries, seeing a p-value of 1/161 hardly eliminates the possibility that this is due to just chance. The p-value of 0.0047, or 1/213, for the adjusted correlation is a bit lower, but it's also more contrived, so we need to adjust our "How many hypotheses are there that would be as, or more, noticeable?" number upwards.

So it's enough that it would be understandable to investigate whether there's something causing a bias, but there's no way to know without empirical investigation, and sometimes things happen just by chance. That's the whole point of a lottery, after all. If you were to win a lottery, you probably wouldn't say "The p-value of me winning is less than 0.05, so obviously the lottery's rigged."