Changing the scale in a hypothesis test of proportions

Question

When performing a hypothesis test of a difference of two proportions, does one change the integrity of the problem when changing the scale?

To clarify: I want to test the difference in proportions of maintenance errors per hour of maintenance for two sets of years, say A = [1990-2000] and B = [2000-2010]. When I did this, I found that I had 4 errors from set A and 4 errors from set B. I also found that I had about 2000 maintenance hours in set A and about 2500 maintenance hours in set B. This does not meet the "rule of thumb" for using the normal distribution in a CI of proportions. Also, I've been told that the n = 2000 is far to greater than x = 4 for the proportion x/n. (Same for y and m).

Are there any general methods to combat this? Does it change the integrity of the problem if I just change the scale from maintenance hours, to days of maintenance? (2000/24 = 83.3 so use 4/83.3 instead of 4/2000)

Sorry if this is unclear, I tried to explain it the best I could. I can't find any literature on this.

Answer 1

You don't really have proportions, you have ratios. A binomial is a certain number of 'successes' out of a fixed number of trials. The easiest way to think about this is a number of heads out of a fixed number of coin flips. Your numerator is an event—analogous to getting a heads—but your denominator isn't a trial. Thus, the binomial distribution is not appropriate for your data.

Your data are counts of events. The simplest distribution for counts is the Poisson. The Poisson distribution is generally taken to be a default distribution for counts, but it is very restrictive and is not actually typically used for that reason. It is much more common to use a distribution that allows for greater flexibility (specifically, greater variance) like the negative binomial. However, you don't have the ability to use the NB, because you have, in essence, only two data. Thus, you are in the unfortunate position of making a very strong (and likely incorrect) assumption, or not being able to test your data.

We also need to account for your denominator, which is an opportunity for an event to occur in some sense, even if it isn't quite a trial. This can be done by using an offset (see here and here).

At this point then, you can conduct a low-powered, possibly invalid test that provides a lower bound on the p-value. Using your data, here is that test (conducted with R):

poisson.test(x=c(4, 4), T=c(2000, 2500))
#   Comparison of Poisson rates
# 
# data:  c(4, 4) time base: c(2000, 2500)
# count1 = 4, expected count1 = 3.5556, p-value = 1
# alternative hypothesis: true rate ratio is not equal to 1
# 95 percent confidence interval:
#  0.232822 6.711136
# sample estimates:
# rate ratio 
#       1.25 

This result shouldn't be that surprising. You have the same number of errors in each case, and nearly the same number of hours.

Answer 2

If you cannot use the approximation of normality, then don't use it. Use some sensible approximation. Maintenance errors per hour sound like they could reasonably be regarded as Poisson distributed. Please check, if this applies.

If so, we have one single parameter $\lambda$ for each group, let'S call them $\lambda_A$ and $\lambda_B$. Assuming a flat prior (which is not that reasonable but a good first approach), we can model these with a conjugate prior as $Gamma$ distributed: https://en.wikipedia.org/wiki/Conjugate_prior#Discrete_distributions

$\lambda_A$ als being $Gamma(4, 2000)$ and $lambda_B$ as being $Gamma(4, 2500)$ distributed.

Let's sample 1 million possible $\lambda_A$ and one million possible $\lambda_B$:

lambda_A <- rgamma(1000000, 4, 2000)
lambda_B <- rgamma(1000000, 4, 2500)

plot(density(lambda_B), col="blue")
points(density(lambda_A), col="red", type="l")

Now let's see, how many times in one million samples. $\lambda_A$ was larger then $\lambda_B$

> sum(lambda_A < lambda_B)/1000000
[1] 0.379699
> sum(lambda_A == lambda_B)/1000000
[1] 0
> sum(lambda_A > lambda_B)/1000000
[1] 0.620301   

So $\lambda_A$ was smaller in 38% of the samples and larger in 62%. This gives you a good estimation of the chances of each being larger then the other one.

How much larger?

> summary(lambda_A - lambda_B)
  Min.    1st Qu.     Median       Mean    3rd Qu.       Max. 
-0.0073480 -0.0004217  0.0003514  0.0004001  0.0011750  0.0095330 
> plot(density(lambda_A - lambda_B))

The Poisson distributions seems more appropriate than normal approximation, except if you have more knowledge about the data generation process, than something different might be superior.

You might want to check, whether one million was a large enough sample to meet your precision needs.

Note: This is invariant on hours or working days:

> lambda_A <- rgamma(1000000, 4, 2000/24)
> lambda_B <- rgamma(1000000, 4, 2500/24)
> sum(lambda_A < lambda_B)/1000000
[1] 0.380063

-- still 38% as you'd expect from a sensible computation.