12

Let's say we are repeatedly tossing a fair coin, and we know number of heads and tails should be roughly equal. When we see a result like 10 heads and 10 tails for a total of 20 tosses, we believe the results and are inclined to believe the coin is fair.

Well when you see a result like 10000 heads and 10000 tails for a total of 20000 tosses, I actually would question the validity of the result (did the experimenter fake the data), as I know this is more unlikely than, say a result of 10093 heads and 9907 tails.

What is the statistical argument behind my intuition?

csgillespie
  • 13,029
Haibao Tang
  • 165
  • 6

6 Answers6

21

Assuming a fair coin the outcome of 10000 heads and 10000 tails is actually more likely than an outcome of 10093 heads and 9907 tails.

However, when you say that a real experimenter is unlikely to obtain an equal number of heads and tails, you are implicitly invoking Bayes theorem. Your prior belief about a real experiment is that Prob(No of heads = 10000 in 20000 tosses | Given that experimenter is not faking) is close to 0. Thus, when you see an actual outcome that the 'No of heads = 10000' your posterior about Prob(Experimenter is not faking | observed outcome of 10000 heads) is also close to 0. Thus, you conclude that the experimenter is faking the data.

  • Very well explained! What a wonderful example to the Bayes theorem approach. – Tal Galili Nov 23 '10 at 18:04
  • 1
    @Srikant: that prior cannot be formally defined. In any case, Prob(No of heads = X | experimenter is not faking) is always around zero when N=20000, no matter the value of X and no matter your prior. So your posterior for any number is also always close to 0. I don't see what this has to do with Bayes theorem. – Joris Meys Nov 25 '10 at 15:29
  • All of this from a guy that was holed up trying to prove god existed. Elegant, really. – Brandon Bertelsen Nov 25 '10 at 16:29
  • @Joris I am talking about the OP's subjective beliefs about Prob(No of heads = 10k | experimenter is not faking) and not about the actual probabilities assuming a fair dice. I would imagine that the OP's subjective beliefs require that Prob(No of heads = 10k | experimenter is faking) be close to 1. Assuming prior probabilities vis-a-vis an experimenter being a liar or not (say Prob(fake experimenter) = 0.5) gives the desired posterior. Does that help clarify? –  Nov 25 '10 at 22:16
  • @Srikant: Point is that getting exactly 10000 out of 20000 is about just as likely as getting exactly 10093 or 9781, being about 0.5^20000. So if you calculate it, P(cheating|10) appx P(not cheating|10) appx 0. In R both are 0 if you use P(10k|cheating) = 0.995, using Bayes theorem. This means by the way that cheaters almost always chose 10k, which is not the assumption OP makes. And on a more philosophical point, you could say that if you believe chance is big he's cheating when heads=10k, chance is bigger he's cheating. Bayes never meant his theorem to be used that way. – Joris Meys Nov 25 '10 at 22:45
  • @Joris No the probabilities are not exactly equal. You need to also account for the combinatorial coefficients. In any case, we may be talking past each other as you seem be talking about probabilities with a throw of fair dice whereas I am talking about subjective probabilities. –  Nov 25 '10 at 23:02
  • @Srikant : appx is not exactly. I am not talking about probabilities with fair dice, I'm calculating posteriors using bayes theorem and your subjective probabilities. Which still makes the posteriors to be essentially almost zero and even a bit smaller. But both posteriors are lower than the chance to win the lottery twice. It's not a philosophical point of view, it's basic math. – Joris Meys Nov 25 '10 at 23:25
  • 1
    Putting this in a more general perspective, the point, with which I agree, is that Bayes theorem is at work here. Specifically there is are alternative likelihoods (corresponding to different generative processes) for cheating and for honest experimenters. Establishing the cheating is posterior inference with respect to the intuitive and therefore woefully underspecified cheater process. – conjugateprior Nov 28 '10 at 16:27
  • Happily one can use just this sort of problem to elicit what people really think is 'random' looks like. There is often, it seems, a rather structured model available for things like a 'random coin flip sequence' (it's got negative auto-correlation, fewer runs, etc.) Some fun research modelling both the structure of these intuitions and inferences based on them is here: http://cocosci.berkeley.edu/tom/papers/random.pdf and in later papers. – conjugateprior Nov 28 '10 at 16:33
  • @Conjugate Prior: Yes, bayes theorem is at work here. But with Bayes theorem, as I calculated for you, both posterior probabilities mentioned are close to zero (smaller than R can cope with actually). So the posterior conclusion, based on Bayes theorem, when you have exactly 10k, is that the researcher is not cheating. Why talking philosophy if you have numbers? – Joris Meys Nov 29 '10 at 01:14
  • @Conjugate Prior. Plus, the paper you refer to mentions that people are likely to introduce less runs when they consider a "random" process (p=0.6 instead of 0.5). So it rejects intuition rather than confirm it. The paper also talks about sequences and patterns, but that is not what is discussed here. HTTHHT is more likely than HHHTTT according to that paper, but both have a p of exactly 0.5 – Joris Meys Nov 29 '10 at 01:20
  • @Joris Meys Your calculations are off, way off. The chance of getting exactly 10,000 heads is 0.564%. The chance of getting exactly 10,093 heads is 0.238%. (These assume any reasonably diffuse and uninformative prior.) Both are astronomically larger than 2^(-20000) and neither is so close to zero as to be entirely surprising even if it were predicted in advance (concerning which see a comment by @G. Jay Kerns following his answer). – whuber Dec 12 '10 at 18:12
  • 1
    @Srikant @whuber: the combinatorials... you're right. I started from an uniform probability, which is off course nonsense in this case. My bad – Joris Meys Dec 12 '10 at 21:34
  • After all these years this answer still bothers me. Perhaps the cause is the casual reference to "close to zero" as sufficient reason to conclude the results were faked. Strong conclusions require strong evidence. Just how "close to zero" is this astonishing result? The Binomial distribution tells us the chance is $\binom{20000}{10000}2^{-20000}.$ This is easy to approximate with wonderful accuracy using Stirling's approximation; it's greater than half a percent: one in 200. Would you, as a statistician, be willing to ruin someone's reputation by ignoring a chance that large? – whuber Jun 07 '22 at 22:14
12

I like Srikant's explanation, and I think the Bayesian idea is probably the best way to approach a problem like this. But here is another way to see it without Bayes: (in R)

dbinom(10, size = 20, prob = 0.5)/dbinom(10000, 20000, 0.5)

which is about 31.2 on my system. In other words, it is over 30 times more likely to see 10 out of 20 than it is to see 10,000 out of 20,000, even with a fair coin in both cases. This ratio increases without bound as the sample size increases.

This is a sort of likelihood ratio approach, but again, in my gut this feels like a Bayesian judgement call more than anything else.

  • 1
    Why the ratio? Why not just state that the probability of that exact draw is extremely low? – Andy W Nov 10 '10 at 17:06
  • 5
    The assertion that a particular probability is low out of context isn't convincing. The probability that I am exactly as tall as my height (whatever that may be) is zero. And, yes, it's problematic to even define height with infinite precision, yada, yada, yada... My point is that the maelstrom of existence churns with events of infinitesimal probability happening all the time! 10,000 out of 20,000 - out of context - doesn't surprise me at all. Regardless of what its numerical probability may be. –  Nov 11 '10 at 00:12
  • Why is an experiment with 20 coin flips even relevant to the present question?? Sure, it's mentioned in the question, but that does not validate it as a reference probability. – whuber Jun 07 '22 at 22:17
10

A subjectivist Bayesian argument is practically the only way (from a statistical standpoint) you could go about understanding your intuition, which is--properly speaking--the subject of a psychological investigation, not a statistical one. However, it is patently unfair--and therefore invalid--to use a Bayesian approach to argue that an investigator faked the data. The logic of this is perfectly circular: it comes down to saying "based on my prior beliefs about the outcome, I find your result incredible, and therefore you must have cheated." Such an illogical self-serving argument obviously wouldn't stand up in a courtroom or in a peer review process.

Instead, we could take a tip from Ronald Fisher's critique of Mendel's experiments and conduct a formal hypothesis test. Of course it's invalid to test a post hoc hypothesis based on the outcome. But experiments have to be replicated to be believed: that's a tenet of the scientific method. So, having seen one result we think might have been faked, we can formulate an appropriate hypothesis to test future (or additional) results. In this case the critical region would comprise a set of results extremely close to the expectation. For instance, a test at the $\alpha$ = 5% level would view any result between 9,996 and 10,004 as suspect, because (a) this collection is close to our hypothesized "faked" results and (b) under the null hypothesis of no faking (innocent until proven guilty in court!), a result in this range has only a 5% (actually 5.07426%) chance of occurring. Furthermore, we can put this seemingly ad hoc approach in a chi-square context (a la Fisher) simply by squaring the deviation between the observed proportion and the expected proportion, then invoking the Neyman-Pearson lemma in a one-tailed test at the low tail and applying the Normal approximation to the Binomial distribution.

Although such a test cannot prove fakery, it can be applied to future reports from that experimenter to assess the credibility of their claims, without making untoward and unsupportable assumptions based on your intuition alone. This is much more fair and rigorous than invoking a Bayesian argument to implicate someone who might be perfectly innocent and just happened to be so unlucky that they got a beautiful experimental result!

whuber
  • 322,774
6

I think your intuition is flawed. It seems you are implicitly comparing a single, "very special" result (exactly 10000 heads) with a set of many results (all "non-special" numbers of heads close to 10000). However, the definition of "special" is an arbitrary choice based on our psychology. How about binary 10000000000000 (decimal 8192) or Hex ABC (decimal 2748) - would that be suspiciously special as well? As Joris Meys commented, the Bayes argument would essentially be the same for any single number of heads, implying that each outcome would be suspicious.

To expand the argument a bit: you want to test a hypothesis ("the experimenter is faking"), and then you choose a test statistic (number of heads). Now, is this test statistic suited to tell you something about your hypothesis? To me, it seems the chosen test statistic is not informative (not a function of a parameter specified as a fixed value in the hypothesis). This goes back to the question what you mean by "cheating". If that means the experimenter controls the coin at will, then this is not reflected in the test statistic. I think you need to be more precise to find a quantifiable indicator, and thus make the question amenable to a statistical test.

caracal
  • 12,009
  • +1, But I'm not convinced. What's special about 10,000 is that it exactly equals the expected number of heads under the hypothesis that the coin is fair. This fact is independent of any psychology or system of number representation. The analysis in this response might provide some insight into a situation where, say, 20,005 coins were flipped and 10,000 heads (and therefore 10,005 tails) were noted and somebody's "intuition" suggested fakery took place. – whuber Dec 12 '10 at 18:18
  • I fully agree that - as you point out in your answer - it all depends on the a-priori definition of the hypothesis: if you define in advance that by "faking the experiment" you mean "achieving a result for number of heads that is close to the expected value", then that's a basis for a statistical test with "number of heads" as a test statistic. However, without such a-priori clarification, the meaning of "faking" and "special value for number of heads" remain cloudy, and it's not clear what they have to do with each other. – caracal Dec 13 '10 at 16:06
  • @whuber I think that just shifts the question to why the number equal to the expected number of heads under the hypothesis that the coin is fair, is special. Isn't the number equal to the expected number of heads under the hypothesis that the coin is fair minus one about just as special? – Paradox Nov 16 '21 at 16:55
  • @Paradox The problem is that every number is "special." This argument sometimes is taken to an extreme: when you model a number as the realization of a continuous random variable, by definition that value had probability zero of appearing. Perhaps another (and more fruitful way) of addressing the immediate question is to point out that the vagueness of the phrase "roughly equal" lends itself to all kinds of invalid arguments, such as the one offered in the second half of the question. – whuber Nov 16 '21 at 17:11
4

The conclusion you draw will be VERY dependent on the prior you choose for the probability of cheating and the prior probability that, given the flipper is lying, x heads are reported.

Putting the most mass on P(10000 heads reported|lying) is a little counter intuitive in my opinion. Unless the reporter is naive, I can't imagine anyone reporting that kind of falsified data (largely for the reasons you mentioned in the original post; it's too suspicious to most people.) If the coin really is unfair and the flipper were to report falsified data, then I think a more reasonable (and very approximate) prior on the reported results might be a discrete uniform prior P(X heads reported|lying) = 1/201 for the integers {9900, ..., 10100} and P(x heads reported|lying) = 0 for all other x. Suppose that you think the prior probability of lying is 0.5. Then some posterior probabilities are:

P(lying|9900 heads reported) = P(lying|10100 heads reported) = 0.70;

P(lying|9950 heads reported) = P(lying|10050 heads reported) = 0.54;

P(lying|10000 heads reported) = 0.47.

Most reasonable numbers of reported heads from a fair coin will result in suspicion. Just to show how sensitive the posterior probabilities are to your priors, if the prior probability of cheating is lowered to 0.10, then the posterior probabilities become:

P(lying|9900 heads reported) = P(lying|10100 heads reported) = 0.21;

P(lying|9950 heads reported) = P(lying|10050 heads reported) = 0.11;

P(lying|10000 heads reported) = 0.09.

So I think the original (and highly rated answer) could be expanded a little bit; in no way should you conclude that the data is falsified without thoroughly considering prior information. Also, just thinking about this intuitively, it seems that the posterior probabilities of lying are likely to be influenced more by the prior probability of lying rather than by the prior distribution of heads reported given that the flipper is lying (except for priors that put all their mass on a small number of heads reported given the flipper is lying, such as in my example.)

HairyBeast
  • 511
  • I think this is a very good answer, but I disagree with your second paragraph. I don't think Srikant's original conditional probability is counter-intuitive, and simply because it is a hard question to answer is not an argument against it. I also don't think your uniform probability of lying within 9900 to 10100 makes any sense at all, although it is useful for demonstration purposes. – Andy W Dec 12 '10 at 23:47
2

For the Bayesian explanation, you need a prior probability distribution on the reported results by a lying coin flipper, as well as a prior probability of lying. When you see a value that is much more likely under the lying distribution than the random flips one, that makes your posterior probability of lying much higher.

internet
  • 131