5

I have a question for setting null hypothesis in binomial test. More specifically, is there any way to assume chance level when it is unknown?

For example, let's say that I have a slightly bent coin. So chance for getting Heads or tails is not equal. Instead, it's more likely that it will give us tail. To find out the probability, I tossed the coin 100 times, and got P(getting tail) = 68%

Then I bent it even more, and I wanted to check if it had any effect on the probability. So I threw it 10 times and got tails 9 times (90%).

Is it correct to use one sample binomial test to check if 9 out of 10 (90%) is significantly different using 68% as expected probability?

I'm not sure if it's okay because it "assumes" that chance level is 68% based on experience of throwing the coin 100 times, instead of mathematical calculation.

Thanks for reading.

Roas Clack
  • 95

3 Answers3

5

No, it is not correct. In this example you have two samples so you should be using a two-sample hypothesis test. There are a few different two-sample binomial tests available, so you will need to choose one.

Ben
  • 124,856
4

Is it correct to use one sample binomial test to check if 9 out of 10 (90%) is significantly different using 68% as expected probability?

I don't think this is correct because you would assume that 68% was measured without uncertainty. Intuitively, 68% obtained from 10000 flips is "better", more certain than 68% from just 100 flips.

In your case you could use Fisher test for the null hypothesis that 68/100 and 9/10 can originate from the same coin. In R:

mat <- matrix(c(68, 9, 100-68, 10-9), nrow= 2)
mat
     [,1] [,2]
[1,]   68   32
[2,]    9    1

fisher.test(mat)

Fisher's Exact Test for Count Data




data:  mat
p-value = 0.2765
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
 0.005230503 1.851113305
sample estimates:
odds ratio 
 0.2384069 
```
dariober
  • 4,250
  • Thanks for the reply. Would there be any chance for using measured data as expected value if the observation number increased? My actual data can't be tested using Fisher Exact Test nor Mcnemar Test, because it is partially paired. – Roas Clack Apr 25 '22 at 09:00
  • @RoasClack I'm not sure I can answer your comment without more detail - perhaps edit the question to better explain your situation. E.g., what do you mean by partially paired? – dariober Apr 25 '22 at 09:56
  • You can see that from my previous question here: https://stats.stackexchange.com/questions/572561/how-to-test-if-watching-a-advertisement-only-1-time-is-enough-to-change-ones-pr?noredirect=1#comment1056520_572561 Thanks for the kind answer. – Roas Clack Apr 25 '22 at 10:06
1

No, you treat the result from your sample of 100 ($\hat{p}=68/100$) as a sample statistic not a population proportion. If you repeated that trial of $100$ before the second bending, you would be very likely not to get $68$ on the second attempt; you'd see $65$ or $74$ or $70$ or $63$, or some other number, and a third trial would be likely different again from the first two.

Which is to say $0.68$ is just a noisy estimate of the true proportion. You should not ignore that uncertainty, treating it as if it were the true proportion.

Glen_b
  • 282,281
  • Thanks for the comment. I see that even though 100 seems relatively large compared to throwing just 10 times, it's still an sample. – Roas Clack Apr 27 '22 at 02:49
  • Going from 10 to 100 trials. you'd only reduce the standard error to about 32% of what it was -- not an especially dramatic reduction. People often underestimate quite how large a sample it would take to reasonably treat the sample proportion as known exactly. Much easier to just calculate it as what it is and not worry. – Glen_b Apr 27 '22 at 02:52
  • Can I ask you another question? What if I predict the population proportion not by sampling, but by using mathematical models like linear regression? More specific question is posted here in case you're interested: https://stats.stackexchange.com/questions/573019/binomial-test-with-chance-level-prediction – Roas Clack Apr 27 '22 at 02:53
  • I'll look. I don't promise that I will have time to answer. – Glen_b Apr 27 '22 at 02:53
  • Thank you so much – Roas Clack Apr 27 '22 at 02:57