1

I am estimating a proportion p and believe it to be 0.8. I want to show that it is almost certainly greater than 0.7. I have done a power calculation by simulation, and I have figured out that I need n samples to be 80% sure that the lower confidence limit for p will be greater than 0.7 (given p is really 0.8) I am using a 90% confidence interval, because this will have 95% of the probability mass greater than the lower limit. So far, so good.

Suppose that I have done the experiments, and I find that this lower confidence limit is not greater than 0.7. This could be due to a) my assumption of 0.8 is wrong or b) bad luck (or both) Now someone says: We'll take another m samples, and see what lower limit we will have then. Suppose that now we are successful, and this lower limit is indeed greater than 0.7.

What do I know now ?

If I would have done n+m experiments to begin with, all would have been good. But now I have been unsuccessful with n samples, and I have chosen m by looking at the results of the first n experiments.

I don't have any data yet, I am just trying to get precise the implications of this strategy. I have seen this post data peeking and increasing sample size but I still don't know the answer.

Old_Mortality
  • 359
  • You say, "If I would have done $n+m$ experiments to begin with, all would have been good. But now I have been unsuccessful with n samples, and I have chosen m by looking at the results of the first n experiments. // Not so. Large enough $n$ for good power to detect $p > .8$ (if true) is nis no guarantee you'll find $p >.8,$ whether true or not. // Strictly speaking, it is "P-hacking" to increase $n$ in mid=experiment having seen some early data values. – BruceET Jul 05 '22 at 22:45
  • Yes I agree. I never meant to say that sufficient power is such a guarantee. I meant to say that if I had decided in advance I wanted n+m samples, and I would have done that, and found what I had hoped for, all would be good. My question is: If I know in advance I am going to do what you describe as 'p-value hacking', how do I interpret the results? – Old_Mortality Jul 05 '22 at 23:21
  • 2
    People have grown suspicious about P-values. If you always start out with a small sample and then based on favorable early returns decide to do a larger sample, that is not the same as always doing a larger sample. Similarly, it wouldn't be the same to stop when the success probability happens to get large. These alternative stopping rules are OK, but need their own critical values. – BruceET Jul 06 '22 at 00:04

2 Answers2

3

Since you haven't started the experiment, I suggest a different approach to determine sample size: choose n to control the accuracy of the estimate, not the power of a statistical test. Here accuracy means the margin of error $\delta$ of the 100(1-α)% confidence interval for the population proportion $p$.

Here is how the process could work out in your case. You believe the true proportion is $p$ = 0.8 or at least $p$ > 0.7. Once you collect data you would be able to construct a 100(1-α)% for $p$:

$$ \begin{aligned} \hat{p} \pm \delta = \hat{p} \pm z_{1-\alpha/2}\sqrt{\hat{p}(1-\hat{p})/n} \end{aligned} $$

You can specify the half-width $\delta$ you'd like to achieve. For example, $\delta$ = 1 is probably too wide because even if $p$ is exactly 0.8 the confidence interval might be centered at a value for $\hat{p}$ that's slightly below 0.8, so the confidence interval will include 0.7. On the other hand, $\delta$ = 0.01 might be unnecessarily precise because you want to check whether the lower limit $\hat{p} - \delta$ is above 0.7. Perhaps $\delta$ = 0.05 sounds about right? Then you compute the required sample size is a function of $\delta$ and the variance of $\hat{p}$. (I also assume the significance level is $\alpha$ = 0.05.)

$$ \begin{aligned} n = z_{1-\alpha/2}^2\hat{p}(1-\hat{p})/\delta^2 \end{aligned} $$

For simplicity, let's plug in 1/4 for the variance of $\hat{p}$. This is the "worst case" value because the function $p(1-p)$ is maximized at $p$ = 1/2 for $p$ between 0 and 1. You estimate, conservatively, that you'll need a sample of size $n$ = 385 (after rounding up).

See the section Sample Size Justification in Improving Your Statistical Inferences by Daniël Lakens.

dipetkov
  • 9,805
  • Thank you for your help. I like this approach, but it does not help me with the two tests (one for the first n samples, and then for the next m) I am thinking I can just apply some correction, like Bonferroni, but the two tests are not independent. I should add it is not my idea to it this way, I am anticipating that is what 'they' are going to do, and am wondering how I will deal with it later. – Old_Mortality Jul 12 '22 at 09:24
  • Why are you planning to do two tests? The comments from @BruceET and the other answer by kqr also suggest to rethink your strategy. – dipetkov Jul 12 '22 at 10:56
  • It's not my idea. This is what I see coming, and then they'll ask me to analyse the data. I am hoping to anticipate how I will do that. But yes, I'll try to persuade my client not to do this. – Old_Mortality Jul 17 '22 at 20:44
2

Not only is this situation common -- some people even deliberately design their studies for two-stage sampling. Cochran mentions in Sampling Techniques (1955) the approach known as Stein's method.

First you do a regular sample and derive the statistic of interest. Then based on the information of the first, you do a second sample designed to be of the right size to get the interval you need.

The method proceeds in much the same way as the answer you have already received from dipetkov, other than the fact that you split it up into two stages.

kqr
  • 814