3

I'm trying to perform an a-priori power analysis for a mixed effects model using the simr package, but I'm not sure which effect size to choose.

When performing a power analysis for ANOVA I've used Cohen's suggested f values to represent a small, large, and medium effect size (0.1, 0.25, and 0.4). It looks like simr uses slopes as effect sizes, so I'm wondering what would the equivalent small/medium/large effect sizes would be in this case.

My study would have two fixed effects (s and z). S has two factors and z has 3. There would be a random intercept (t) as well.

Thanks for the help!

mels
  • 57
  • 2
    Hi @mels! I suggest you think about what "small", "medium", and "large" means in the context of your study. Cohen just made up those values anyway. There's no right answer to this. E.g. in a behaviour change study, maybe a 5% change is large. But in some other context, maybe 50% change is large. – Alex J Feb 28 '24 at 23:47

1 Answers1

5

I have struggled with this same issue, and it is my understanding that it's not possible, or at least it's very difficult to define standardized effect sizes in multilevel models.

See, for instance, this thread and this thread and this discussion and this article:

Rights, J. D., & Sterba, S. K. (2019). Quantifying explained variance in multilevel models: An integrative framework for defining R-squared measures. Psychological Methods, 24(3), 309–338. https://doi.org/10.1037/met0000184

The article focuses on R-squared measures but the reasoning is the same as it is for other effect size measures.

This is indeed problematic when you want to run power/sample size calculations for a mixed model. I have addressed the issue in two ways:

  1. find a previous study (or a couple of studies, if possible) using the same scales as I will be using and use their (unstandardized) effect size as the target effect size in power analysis in simr or

  2. refer to existing simulation studies such as this:

Arend, M. G., & Schäfer, T. (2019). Statistical power in two-level models: A tutorial based on Monte Carlo simulation. Psychological Methods, 24(1), 1–19. https://doi.org/10.1037/met0000195

see also this paper:

Kumle, L., Võ, M.LH. & Draschkow, D. Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R. Behav Res 53, 2528–2543 (2021). https://doi.org/10.3758/s13428-021-01546-0

when determining sample size for my upcoming study and skip my own sample size calculations.

Also, as Alex J said, even with standardized effect sizes, these things are not exact and there are no clear answers. E.g. with multilevel models, even if you had the standardized effect size, you need to decide the size of the random intercept, random slope, their covariance, residual correlation structure - lots of moving parts that can affect the power estimate and for which there are no clear guidelines.

Edited to add: though Westfall et al. (2014) present a way of calculating Cohen's d for a mixed/multilevel model:

Westfall, J., Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143(5), 2020

I don't know whether this way is commonly used, I only now found the article.

Sointu
  • 1,846