I'm performing a DoD (difference of differences) hypothesis test, using a Bayesian model. I analyzed the data post-experiment and found that the mean posterior lift in metric attributable to treatment exposure was 1.53% and the proportion of the posterior > 0 was 94% (94% chance the treatment had a real effect on metric.)
A project manager asked me to re-analyze the experiment using a subset of data. He believed that some samples were 'more representative' of treatment exposure.
I did so and opened a real can of worms. The trend reversed. So now the question is, what minimum sample size do we need for the posterior to be credibly greater than 0.
I'm curious what power analysis analogs are available?
In my use case, the exact samples used is quite noisy. For example, we switched from 38 individuals to 19, inducing the trend reversal. We're keen to know-- at what sample size is so large that a trend reversal is highly unlikely?
The obvious answer is that my sample size is simply too small. But I need to know by how much and why so. I don't think that a heuristic would be very persuasive in my organization.
I've read that Bayes Factor is applicable- but isn't that a comparison of priors? Here I'm interested in the likelihood and posterior as well.
I'm considering using a bootstrap scheme to sample x individuals from each group, compute the posterior and record whether it was > 0 (or not.) Then consider the average of this binary outcome as 'coherence'. If I repeat this for varying sample sizes, I suspect that an elbow plot will emerge that would guide me to a reasonably choice of n.
But this feels handwavy rather than proof driven.
