2

I used simr package in R to determine the minimum number of subjects I need to obtain the main effect of variable A and the interaction between variable A and B on a number of dependent variables: DV1 and DV2.

I ran the power analysis based on a dataset of 20 subjects using linear mixed effect model. However, the PowerCurve results give very low power for the main effect of variable A on DV1 (see the image below) even when I increased the subjects to 100 based on 100 simulations.

DV1 power analysis

Does it mean that there is very low chance that I can obtain the main effect of variable A on DV1 in my final dataset, and therefore I should not actually look at DV1? When I ran the powercurve on DV2, it indicates that I need 60 subjects to obtain the main effect of variable A with 80% power. Can I use this number as the minimum number of my sample size? And does the number of simulations matter in the power given by the powercurve?

mkt
  • 18,245
  • 11
  • 73
  • 172
Chloe
  • 363
  • "I ran the power analysis based on a dataset of 20 subjects..." Was this from a pilot experiment? If it was a published (and not pre-registered) analysis, it's possible that you used an inflated estimate of the real effect size. We now know that unfortunately, a large proportion of such experiments in the social sciences were (are?) underpowered. – mkt Sep 14 '22 at 06:08
  • If you need to look at DV1, DV2 and their interaction (which I assume are experimental manipulations), it's quite plausible that the sample size you will need will run into the thousands. If that is not feasible because of experimental constraints, then it makes sense to focus on DV2. – mkt Sep 14 '22 at 06:10
  • @mkt thanks for your comments. No this experiment has not yet been published or pre-registered. I used 20 subjects to fit a lmer model to determine what fixed effect I should use in building an artificial model for the simulation. The fixed effect of variable A that I observed from 20 subjects is actually quite small (e.g., 0.005), so I guess that is the power curve results indicate that I need quite a large sample size to obtain that effect with 80% power. – Chloe Sep 14 '22 at 07:55

1 Answers1

1

Here's the problem: the estimated effect sizes you used for your power analysis are likely to be highly noisy because they come from a small dataset of 20 subjects. So much so that you probably shouldn't have much faith in either the low estimate for DV1 or the higher estimate for DV2. You may be on more solid ground if you have additional lines of evidence pointing towards those effect sizes, though you should be aware that published estimates are likely to be biased upwards.

But you have to take some decision, and using your pilot and power analysis is certainly a good start. What other information can you use? Well, I'd strongly recommend considering what the minimum interesting/meaningful effect size would be, and calculating what sample size would give you 90% - 95%% power for that effect size. The choice of threshold is arbitrary but 80% power seems low to me. Even if everything else is accurate, that's still a rather high 1 in 5 chance of failing to detect the effect of interest.

Be aware that with reasonable assumptions, estimating interactions reliably may need >10X as much data as the main effects. So it may make sense to focus on the main effects if you are logistically constrained.

I can't provide strong advice about whether to drop DV1 entirely, because that depends on your interests, constraints and the nature of the data (I assume experimental). But based on the limited information you present, it seems reasonable to focus on DV2. Note that based on what I said earlier, it would still be advisable to use a sample size that is more than the minimum your power analysis suggests. In general, I think reliable parameter estimates in these kinds of studies require much higher sample sizes than a few tens of individuals, though of course this varies substantially depending on the question.

As for the number of simulations: yes, 100 seems low. Given that there's little cost to running longer simulations, I would try at least 10-100 times as many.

mkt
  • 18,245
  • 11
  • 73
  • 172
  • thanks @mkt. In terms of the selection of the minimum interesting/meaningful effect size, would you recommend using the Cohen's d value and converting it into the fixed effect estimate? I just came across an article that suggests that an odd ratio of 1.68 can be translated into a small effect size of d=0.2 (which is what I expect from my study). Therefore, the fixed effect estimate can be set as 0.52 (with its odd ratio being 1.68), although 0.52 is much larger than the estimated given by my current model. – Chloe Sep 15 '22 at 02:08
  • Here is the article: Chen, Henian, Patricia Cohen, and Sophie Chen. 2010. “How Big Is a Big Odds Ratio? Interpreting the Magnitudes of Odds Ratios in Epidemiological Studies.” Communications in Statistics–Simulation and Computation 39 (4): 860–64 – Chloe Sep 15 '22 at 02:08
  • @Chloe Rather than going based on a general Cohen's d recommendation, I suggest instead that you define for your problem/domain what would be a meaningful minimally interesting effect size. – mkt Sep 15 '22 at 06:37
  • thanks @mkt Do you think it is appropriate to use the same random effect variances from model fitted by the pilot data when generating a new model for power simulatio？It seems to be hard to find the appropriate random effect structure from literature especially given the novelty of my study. – Chloe Sep 19 '22 at 01:49
  • @Chloe That sounds fine but I'm not sure. I suggest posting that as a new question. – mkt Sep 19 '22 at 05:54