2

I'm experiencing an hard time looking for the proper analysis to use to compute the power for an experiment i'd like to implement, so here is my question for you.

I'd like to run a quasi-experimental design A (manipulated: 3 levels) X B (measured: ordinal with 5 levels but treated as continuous) and controlling for 2 continuous confounding variables. The instruments I'm using are quite reliable (alpha's are .84, .94, .99). My first choice would be simply running on G*Power a power analysis for a multiple regression with 3 tested predictors (IV,IV and interaction term) and 5 predictors in total (IV,IV,interaction term and 2 confounders), with a small predicted interaction effect.

But I know that there are many things wrong here. I think that treating and ordinal IV as continuous is the worst, cause therein lies the problem: if I had 5 levels I would need 1829 subjects (Ancova, fixed effects, main effects and interactions: EF f=.10, alpha = .05, Power = .80, df = 14, groups = 15, covariates = 2).

Then I think about the second option. Theory on the measured IV states that answers from 4 and above are to be treated as 'high' so, i think, I could split subjects on this premise. But then i could lose information and/or power, and, moreover, I would anyway need more than 1200 subjects.

Then, i think moreover (third possibility), I could just feed parameters into a hierarchical regression and see what sample size I get but the manipulated variable obviously doesn't allow me to use this stream of thought.

The main problem of this design is, I think, the measured predictor.

Any suggestion about how to calculate the sample size will be wholeheartedly appreciated.

Thanks all.

Edit: To play around with InteractionPoweR, following some good advice in the comments, here are some numbers:

library(InteractionPoweR)
power_test = power_interaction_r2(
alpha = 0.05,                  # p-value
N = seq(250,300, by=50),       # sample size
r.x1.y = .23,                   # correlation between x1 and y
r.x2.y =  .23,                  # correlation between x2 and y
r.x1.x2 = .53,                  # correlation between x1 and x2
r.x1x2.y = seq(.1,.2,by=.001))  # correlation between x1x2 and y
power_test
#result = sample size = 300; correlation between x1x2 and y= 0.169; power= 0.8074179
#result = sample size = 250; correlation between x1x2 and y= 0.184; power=0.8013529

When  r.x1.x2 = 0


power_test = power_interaction_r2(
alpha = 0.05,                  # p-value
N = seq(250,300, by=50),                      # sample size
r.x1.y = .23,                   # correlation between x1 and y
r.x2.y =  .23,                  # correlation between x2 and y
r.x1.x2 = 0,                  # correlation between x1 and x2
r.x1x2.y = seq(.1,.2,by=.001))  # correlation between x1x2 and y
power_test
#result = sample size = 300; correlation between x1x2 and y= 0.155; power= 0.8039252
#result = sample size = 250; correlation between x1x2 and y= 0.170; power=0.8035689

The calculations using the methods here is hard for me to use because actually, I understand, I should have the means and the variances, but i don't have them. Finally, the setup I just tried seems to me just a multiple regression with an interaction term power analysis. I like the fact that with 250 people I'm able to detect such a small effects, but to me it seems maybe too big (especially when i see this). Tomorrow I'll some other solution. Meanwhile, any hint is welcome

Jeremy Miles
  • 17,812
GT87
  • 143
  • 1
    To be honest, I find the sample sizes you get quite realistic. However, I think GPower doesn't allow you to specify main effects or correlations between main effects, which may affect the calculations. Have you tried R's package InteractionPoweR? https://dbaranger.github.io/InteractionPoweR/ This package seems more flexible than GPower. – Sointu Jan 20 '24 at 13:26
  • @Sointu i quote from the cited article "First, it incorporates the correlation between the interacting variables, resulting in more accurate power estimates and frequently greater power than estimates that assume variable independence (Shieh, 2010). Second, it incorporates variable reliability, which can dramatically affect power (Brunner & Austin, 2009). Third, variables can be noncontinuous (i.e., binary or ordinal)." I could use It but i don't know what to do because manipulations in principle are not correlated with measured IVs – GT87 Jan 20 '24 at 14:36
  • 1
    You can specify those correlations as zero. However, if your 5-level IV is non-ordinal (ie. nominal), you'd need to break it down into 4 binary dummies, which may be cumbersome. Alternatives that might work Superpower package's https://cran.r-project.org/web/packages/Superpower/vignettes/intro_to_superpower.html ANCOVA method: https://cran.r-project.org/web/packages/Superpower/vignettes/ANCOVAs.html#ref-shieh_ancova – Sointu Jan 20 '24 at 15:55
  • 1
    @ Sointu Well i'm gonna try it then. And about the IV i think I'll treat it as continuous, because it is the result of a 4 item measure and, as far as i know, in Social psych the very majority of studies treat ordinal variables as continuous, when it is sound. When I'm done i'll post the code. Thank you very much – GT87 Jan 20 '24 at 16:37

1 Answers1

0

The calculation to implement a power analisys for an ANCOVA is quite burdensome (see the links in the question) without the necessary and adeguate literature data and programming, conceptual and statistical skills. In my case, I am fine with what I found with the script above using InteractionPoweR.

In an experiment the manipulated IV is, in principle, uncorrelated with other IV's and confounders or covariates. So I set the r.x1.x2 to 0, as Soint suggested.

Moreover, I know the effect size for both the predictors (r .23 and r.23) because I found them in the literature. Moreover, I now that the reliabilities of my instruments is very high (1,.99, 1). The first coefficient is the manipulation , the literature says it works. The second is my measured IV, as seen in the literature. The third is the VD, a behavioural measure. With this assumptions, I am ok with this power analisys but, for cautionary reasons, I will sample more subjects (+100) because discretizing a continuous variable results in losing power.

library(InteractionPoweR)
power_test = power_interaction_r2(
alpha = 0.05,                  # p-value
N = seq(250,400, by=50),        # sample size
r.x1.y = .23,                   # correlation between x1 and y
r.x2.y =  .23,                  # correlation between x2 and y
r.x1.x2 = 0,                    # correlation between x1 and x2
r.x1x2.y = seq(.1,.2,by=.001),  # correlation between x1x2 and y
rel.x1 = 1,                    # x1 reliability    
rel.x2 =  .99,                  # x2 reliability          
rel.y  =  1)                    # y reliabilty
power_test

Sampler r   power
400    0.134 0.8000285
350    0.144 0.8034047
300    0.156 0.8047891
250    0.171 0.8039929

Finally, I have two very good confounders that should do their job properly, as long as they behave as the literature says.

I'm put. Thanks all

GT87
  • 143