12

My understanding is that a power analysis is post hoc if and only if it uses the observed effect size as the target population effect size.

user1205901 - Слава Україні
  • 12,873

2 Answers2

7

In my field I see people doing post-hoc power analyses when the purpose of the paper is to show that some effect that one might have expected to be present (either because of previous literature, common sense, etc) is not, at least according to some significance test.

However, in these situations, the researcher is in a bit of a bind -- he or she may have obtained a non-significant result either because the effect really is not present in the population or because the study was not sufficiently powered to detect the effect even if it were present. The purpose of the power analysis, then, is to show that, given even a trivially small effect in the population, the study would have had a high probability of detecting that effect.

For a concrete example of this use of post-hoc power analysis, see this linked paper.

Patrick S. Forscher
  • 3,242
  • 1
    That certainly sounds reasonable. Based on your answer I would conclude that there is sometimes a good reason to do post hoc power analyses. That is unless there is some superior method of showing that given even a trivially small population effect a study would have a high probability of detecting that effect. Do you know of any such method? – user1205901 - Слава Україні Sep 30 '13 at 00:19
  • I think that method is precisely a post-hoc power analysis. I suppose one alternative method might be the use of Bayesian methods instead of Pearsonian hypothesis tests, but in my field (psychology), Pearsonian hypothesis testing is still the dominant statistical paradigm. – Patrick S. Forscher Sep 30 '13 at 01:02
  • There's a huge problem with the described approach. Means are always different because of sampling variation, so virtually, any test would be able to detect even a trivially small effect given a large sample (increase your n to 99999999999 and everything could be significant). Also, in the case of a rejected hypothesis, I'm not pretty sure but it's likely that the "obtained power" will be <0.5 always (or, most times, at least). So, it'd always lead to the conclusion that the sample wasn't enough. – Ágatha Nov 09 '18 at 08:19
  • Bruno, your statement is not true unless the population effect is non-zero. If the population effect is zero, then yes, you will get tiny fluctuations in your observed effect, but they will be small, and completely described by the sampling distribution of the parameter of interest, leading to a significant effect at a rate determined by $\alpha$ – Patrick S. Forscher Nov 09 '18 at 18:56
4

You can always compute the probability that a study would have produce a significant result for a given a priori effect size. In theory, this should be done before a study is conducted because there is no point in carrying out a study with low power that has a low chance to produce a significant result when an effect is present. However, you can also compute power after the study to realize that a study had low power or, unlikely, high power to detect even a small effect.

The term post-hoc or observed power is used for power-analysis that use observed effect sizes in a sample to compute power under the assumption that the observed effect size is a reasonable estimate of the true effect size. Many statisticians have pointed out that observed power in a single study is not very informative because effect sizes are not estimated with sufficient precision to be informative. More recently, researchers have started to examine observed power for a set of studies to examine how powerful studies are on average and whether studies report more significant results than the actual power of studies would justify.

https://replicationindex.wordpress.com/tag/observed-power/

Dr. R
  • 51
  • So, @Dr-r, how could someone refer to the first mentioned kind of study? Is there a correct name for that? I have used the "post hoc" function of G*Power, yet I have used the a priori effect size. The reason I'm doing it is that, at first, I have planned using a "guessed" mean difference and a "guessed" standard deviation, and they did differ a lot from the obtained. Also, I couldn't achieve my planned sample size in both groups. I don't want to use the "post hoc" term in my paper because people could get it wrong. So, do you have any suggestion? – Ágatha Nov 09 '18 at 08:11