0

This may be a dumb question but will ask here:

We launched an experiment that has 3 groups Treatment A vs Treatment B vs Control measuring a metric clicks per user.

We did our power analysis before we launched to calculate our sample size required, however, due to a lot of things that happened, we've had to alter our metric quite a bit (how we define the metric) and so i'm being asked to re-calculate this mid experiment

The end goal really is to figure out - "how many days of data do we need" given that we are mid experiment and are on a shortened time line

What is the proper way to approach this? It seems like i can make a lot of statistical errors here so just wanted to get some suggestions and thoughts

jc315
  • 71
  • There's nothing to prevent you from redoing the power analysis, especially since you have preliminary data. Simply use the data you have for your guesses about the standardized effect size and proceed as before. Just don't get so caught up in this that you erroneously use these power analyses repeatedly to determine when to stop the experiment! If it's not very clear why that's wrong, see (for instance) https://stats.stackexchange.com/questions/310119 or https://stats.stackexchange.com/questions/20676. – whuber Jun 16 '23 at 19:15
  • thank you for the response. would the proper way to do this be to take the average baseline of my control group from my experiment? or should i take the baseline of all users at some past point in time, let's say 1 month prior to my experiment running? the values for some reason are quite different – jc315 Jun 20 '23 at 05:57

0 Answers0