How to conduct inference on binomial proportions with small sample sizes?

Question

Suppose we have a one-sample binomial proportions problem with a small number of trials (say 5–10 trials). I want to conduct inference and come up with a p-value for testing $p\neq p_0$ for some $p_0$ prescribed proportion. What are some ways to do this? Is there a Fisher's Exact test analogue for one-sample? What about re-randomization tests?

Answer 1

Just use the binomial distribution directly. Evaluating one-sided p-values is just a matter of binomial tail probabilities. If you observe $y$ successes out of $n$ trials and you want to test $H_0$: $p=p_0$ vs $H_a$: $p>p_0$ then the one-sided p-value is $P(Y \ge y)$ where $Y$ follows the Bin($n$, $p_0$) binomial distribution. The one-sided p-value of $H_0$ vs $H_a$: $p<p_0$ would be $P(Y \le y)$.

For example, suppose $n=8$, you observe $y=7$ and $p_0=0.4$. Then the one-sided upper-tail p-value is

> y <- 7
> n <- 8
> p0 <- 0.4
> pbinom(y-0.5, size=n, prob=p0, lower.tail=FALSE)
[1] 0.00851968

which is 0.0085. This is an exact binomial-test p-value.

Two-sided tests of $H_0$: $p=p_0$ vs $H_a$: $p\ne p_0$, are a little more complicated because there are several competing ways to construct the rejection region when the null distribution is asymetric. If $p_0=0.5$, then the null distribution is symmetric and you can simply multiply the smaller of the two one-sided p-values by 2. If the null distribution is not symmetric then you can still multiply the smallest one-sided p-value by 2 to get a valid two-sided p-value, an approach that is simple and easy but not the most popular as it is somewhat conservative. The method that is most closely analogous to Fisher's exact test is to take the sum of all the binomial probabilities that are less than or equal to $P(Y=y)$ given that $p=p_0$.

My function exactTest implements four different ways to compute the two-sided p-value (see https://rdrr.io/bioc/edgeR/man/exactTest.html). The method of doubling the smallest one-sided p-value is called "doubletail" and the method of adding up the smallest probabilities is called "smallp". Each method corresponds to a different rejection region. If $p_0=0.5$, so the null distribution is symmetric, then all four methods are the same. My function assumes negative binomial counts but reduces to binomial tests when dispersion=0.

The binomial test is also implemented in the binom.test function in R. It uses the "smallp" method for two-sided probabilities, which is the same method used by Fisher's exact test to construct a two-sided rejection region.

There have been many related answers on this forum:

• Exact Binomial Test p-values? (explains the "smallp" method)
• How to properly calculate the exact p-value for binomial hypothesis testing (e.g. in Excel)?
• p-values different for binomial test vs. proportions test. Which to report?
• How do you calculate an exact two-tailed P-value using binomial distribution? (AdamO asserts the "doubletail" method. whuber asserts there are at least five different reasonable ways to define the two-sided rejection region.)
• Why are p-values from the binomial test in R non-monotonic in trials?
• Understanding 2-sided p-values in a binomial distribution

Finally, note that the binomial test is exactly equivalent to a re-randomization test. If you conduct a one-sided randomization test infinitely many times, you'll just get the one-sided binomial p-values.

Answer 2

Rather than doing a hypothesis test, it may be more informative to derive a confidence interval for the probability parameter using the Wilson score interval. You can do this using the CONF.prop function in the stat.extend package. Here is an example where we generate a 95% confidence interval a small amount of data:

stat.extend::CONF.prop(alpha = 0.05, n = 10, sample.prop = 4/10)
    Confidence Interval (CI) 


95.00% CI for proportion parameter for infinite population 
Interval uses 10 binary data points with sample proportion = 0.4000
[0.168180329706236, 0.687326230266342]