3

I have some existing data, and I want to do a power analysis for a second confirmatory study. I want to estimate the number of participants I would need for having a power of .9, keeping the same items I already have. To give a bit of extra context, I am analyzing my data with linear mixed effects models; now I have a huge dataset where the t value for the predictor of interest is 25, so I think I will need less data for my second study.

The standard approach in simulation-based power analysis as I understand it is (e.g. see simr):

  1. Select a number of participants N
  2. Simulate the response variable for some subset of the data with N participants
  3. Refit the model on the simulated data
  4. Calculate the power for N participants as number of simulations that yielded significant results

I was wondering, why do we need to simulate the response data? Why can't I just:

  1. Select a number of participants N
  2. Sample N1 possible subsets of our data with N participants
  3. Refit our mixed effects models N1 times on downsampled data (as opposed to simulated data)
  4. Calculate the power for N participants as the ratio of models in N1 that are significant

Also, in case this second approach makes sense, I was wondering whether I should sample with or without replacement.

a_gdevr
  • 137

1 Answers1

2

In your case, you already have pilot data, and apparently quite a lot of it. In this situation, I would think that your proposed approach is perfectly valid, and will likely enough be quite close to a simulation approach.

Simulations are necessary if you don't have pilot data, but hypothesize an effect that you have not yet observed. That is actually quite a tricky thing, because the simulation presupposes that we get the data generating process right, which is very often a heroic assumption. Another advantage of pilot data!

As to whether to resample with or without replacement: on the one hand, I would do so without replacement, since you would presumably also use every observation only once in your analysis, or to put it another way, you would only measure each unit of measurement once. On the other hand, in a situation with an apparently large sample size of the original pilot study, and a likely much smaller $N$ than that, it will likely not make all that much of a difference, since very few observations will be duplicated in this situation.

Stephan Kolassa
  • 123,354
  • Interesting point about the data generating process (DGP), Stephan. We also have the problem of not knowing the DGP in observational data. We can propose a DGP using something like a DAG, but it's just an informed guess. I think of simulation as an act of creating a DGP. What you create is likely much simpler than the real world, but you do know how the simulated data was generated. – Erik Ruzek Oct 06 '23 at 13:25
  • @ErikRuzek: yes, but if the whole point of our simulation is a sample size determination, i.e., deciding on a sample size we will run an experiment with, your entire logic may break down if what we simulated some DGP that is far away from what reality serves up later... – Stephan Kolassa Oct 06 '23 at 19:20