Are effect sizes really superior to p-values?

Question

Lots of emphasis is placed on relying on and reporting effect sizes rather than p-values in applied research (e.g. quotes further below).

But is it not the case that an effect size just like a p-value is a random variable and as such can vary from sample to sample when the same experiment is repeated? In other words, I'm asking what statistical features (e.g., effect size is less variable from sample to sample than p-value) make effect sizes better evidence-measuring indices than p-values?

I should, however, mention an important fact that separates a p-value from an effect size. That is, an effect size is something to be estimated because it has a population parameter but a p-value is nothing to be estimated because it doesn't have any population parameter.

To me, effect size is simply a metric that in certain areas of research (e.g., human research) helps transforming empirical findings that come from various researcher-developed measurement tools into a common metric (fair to say using this metric human research can better fit the quant research club).

Maybe if we take a simple proportion as an effect size, the following (in R) is what shows the supremacy of effect sizes over p-values? (p-value changes but effect size doesn't)

binom.test(55, 100, .5)  ## p-value = 0.3682  ## proportion of success 55% 

binom.test(550, 1000, .5) ## p-value = 0.001731 ## proportion of success 55%

Note that most effect sizes are linearly related to a test statistic. Thus, it is an easy step to do null-hypothesis testing using effect sizes.

For example, t statistic resulted from a pre-post design can easily be converted to a corresponding Cohen's d effect size. As such, distribution of Cohen's d is simply the scale-location version of a t distribution.

The quotes:

Because p-values are confounded indices, in theory 100 studies with varying sample sizes and 100 different effect sizes could each have the same single p-value, and 100 studies with the same single effect size could each have 100 different values for p-value.

or

p-value is a random variable that varies from sample to sample. . . . Consequently, it is not appropriate to compare the p-values from two distinct experiments, or from tests on two variables measured in the same experiment, and declare that one is more significant than the other?

Citations:

Thompson, B. (2006). Foundations of behavioral statistics: An insight-based approach. New York, NY: Guilford Press.

Good, P. I., & Hardin, J. W. (2003). Common errors in statistics (and how to avoid them). New York: Wiley.

Answer 1

The advice to provide effect sizes rather than P-values is based on a false dichotomy and is silly. Why not present both?

Scientific conclusions should be based on a rational assessment of available evidence and theory. P-values and observed effect sizes alone or together are not enough.

Neither of the quoted passages that you supply is helpful. Of course P-values vary from experiment to experiment, the strength of evidence in the data varies from experiment to experiment. The P-value is just a numerical extraction of that evidence by way of the statistical model. Given the nature of the P-value, it is very rarely relevant to analytical purposes to compare one P-value with another, so perhaps that is what the quote author is trying to convey.

If you find yourself wanting to compare P-values then you probably should have performed a significance test on a different arrangement of the data in order to sensibly answer the question of interest. See these questions: p-values for p-values? and If one group's mean differs from zero but the other does not, can we conclude that the groups are different?

So, the answer to your question is complex. I do not find dichotomous responses to data based on either P-values or effect sizes to be useful, so are effect sizes superior to P-values? Yes, no, sometimes, maybe, and it depends on your purpose.

Answer 2

In the context of applied research, effect sizes are necessary for readers to interpret the practical significance (as opposed to statistical significance) of the findings. In general, p-values are far more sensitive to sample size than effect sizes are. If an experiment measures an effect size accurately (i.e. it is sufficiently close to the population parameter it is estimating) but yields a non-significant p-value then, all things being equal, increasing the sample size will result in the same effect size but a lower p-value. This can be demonstrated with power analyses or simulations.

In light of this, it is possible to achieve highly significant p-values for effect sizes that have no practical significance. In contrast, study designs with low power can produce non-significant p-values for effect sizes of great practical importance.

It is difficult to discuss the concepts of statistical significance vis-a-vis effect size without a specific real-world application. As an example, consider an experiment that evaluates the effect of a new studying method on students' grade point average (GPA). I would argue that an effect size of 0.01 grade points has little practical significance (i.e. 2.50 compared to 2.51). Assuming a sample size of 2,000 students in both treatment and control groups, and a population standard deviation of 0.5 grade points:

set.seed(12345)
control.data <- rnorm(n=2000, mean = 2.5, sd = 0.5)
set.seed(12345)
treatment.data <- rnorm(n=2000, mean = 2.51, sd = 0.5)
t.test(x = control.data, y = treatment.data, alternative = "two.sided", var.equal = TRUE) 

treatment sample mean = 2.51

control sample mean = 2.50

effect size = 2.51 - 2.50 = 0.01

p = 0.53

Increasing the sample size to 20,000 students and holding everything else constant yields a significant p-value:

set.seed(12345)
control.data <- rnorm(n=20000, mean = 2.5, sd = 0.5)
set.seed(12345)
treatment.data <- rnorm(n=20000, mean = 2.51, sd = 0.5)
t.test(x = control.data, y = treatment.data, alternative = "two.sided", var.equal = TRUE)  

treatment sample mean = 2.51

control sample mean = 2.50

effect size = 2.51 - 2.50 = 0.01

p = 0.044

Obviously it's no trivial thing to increase the sample size by an order of magnitude! However, I think we can all agree that the practical improvement offered by this study method is negligible. If we relied solely on the p-value then we might believe otherwise in the n=20,000 case.

Personally I advocate for reporting both p-values and effect sizes. And bonus points for t- or F-statistics, degrees of freedom and model diagnostics!

Answer 3

I currently work in the data science field, and before then I worked in education research. While at each "career" I've collaborated with people who did not come from a formal background in statistics, and where emphasis of statistical (and practical) significance is heavily placed on the p-value. I've learned include and emphasize effect sizes in my analyses because there is a difference between statistical significance and practical significance.

Generally, the people I worked with cared about one thing "does our program/feature make and impact, yes or no?". To a question like this, you can do something as simple as a t-test and report to them "yes, your program/feature makes a difference". But how large or small is this "difference"?

First, before I begin delving into this topic, I'd like to summarize what we refer to when speaking of effect sizes

Effect size is simply a way of quantifying the size of the difference between two groups. [...] It is particularly valuable for quantifying the effectiveness of a particular intervention, relative to some comparison. It allows us to move beyond the simplistic, 'Does it work or not?' to the far more sophisticated, 'How well does it work in a range of contexts?' Moreover, by placing the emphasis on the most important aspect of an intervention - the size of the effect - rather than its statistical significance (which conflates effect size and sample size), it promotes a more scientific approach to the accumulation of knowledge. For these reasons, effect size is an important tool in reporting and interpreting effectiveness.

It's the Effect Size, Stupid: What effect size is and why it is important

Next, what is a p-value, and what information does it provide us? Well, a p-value, in as few words as possible, is a probability that the observed difference from the null distribution is by pure chance. We therefore reject (or fail to accept) the null hypothesis when this p-value is smaller than a threshold ($\alpha$).

Why Isn't the P Value Enough?

Statistical significance is the probability that the observed difference between two groups is due to chance. If the P value is larger than the alpha level chosen (eg, .05), any observed difference is assumed to be explained by sampling variability. With a sufficiently large sample, a statistical test will almost always demonstrate a significant difference, unless there is no effect whatsoever, that is, when the effect size is exactly zero; yet very small differences, even if significant, are often meaningless. Thus, reporting only the significant P value for an analysis is not adequate for readers to fully understand the results.

And to corroborate @DarrenJames's comments regarding large sample sizes

For example, if a sample size is 10 000, a significant P value is likely to be found even when the difference in outcomes between groups is negligible and may not justify an expensive or time-consuming intervention over another. The level of significance by itself does not predict effect size. Unlike significance tests, effect size is independent of sample size. Statistical significance, on the other hand, depends upon both sample size and effect size. For this reason, P values are considered to be confounded because of their dependence on sample size. Sometimes a statistically significant result means only that a huge sample size was used. [There is a mistaken view that this behaviour represents a bias against the null hypothesis. Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples? ]

Using Effect Size—or Why the P Value Is Not Enough

Report Both P-value and Effect Sizes

Now to answer the question, are effect sizes superior to p-values? I would argue, that these each serve as importance components in statistical analysis that cannot be compared in such terms, and should be reported together. The p-value is a statistic to indicate statistical significance (difference from the null distribution), where the effect size puts into words how much of a difference there is.

As an example, say your supervisor, Bob, who is not very stats-friendly is interested in seeing if there was a significant relationship between wt (weight) and mpg (miles per gallon). You start the analysis with hypotheses

$$ H_0: \beta_{mpg} = 0 \text{ vs } H_A: \beta_{mpg} \neq 0 $$

being tested at $\alpha = 0.05$

> data("mtcars")
> 
> fit = lm(formula = mpg ~ wt, data = mtcars)
> 
> summary(fit)

Call:
lm(formula = mpg ~ wt, data = mtcars)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.5432 -2.3647 -0.1252  1.4096  6.8727 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  37.2851     1.8776  19.858  < 2e-16 ***
wt           -5.3445     0.5591  -9.559 1.29e-10 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.046 on 30 degrees of freedom
Multiple R-squared:  0.7528,    Adjusted R-squared:  0.7446 
F-statistic: 91.38 on 1 and 30 DF,  p-value: 1.294e-10

From the summary output we can see that we have a t-statistic with a very small p-value. We can comfortably reject the null hypothesis and report that $\beta_{mpg} \neq 0$. However, your boss asks, well, how different is it? You can tell Bob, "well, it looks like there is a negative linear relationship between mpg and wt. Also, can be summarized that for every increased unit in wt there is a decrease of 5.3445 in mpg"

Thus, you were able to conclude that results were statistically significant, and communicate the significance in practical terms.

I hope this was useful in answering your question.

Answer 4

The utility of effect sizes relative to p-values (as well as other metrics of statistical inference) is routinely debated in my field—psychology—and the debate is currently “hotter”, than normal for reasons that are relevant to your question. And though I am sure psychology isn’t necessarily the most statistically sophisticated scientific field, it has readily discussed, studied—and at times, demonstrated—limitations of various approaches to statistical inference, or at least how they are limited by human use. The answers already posted include good insights, but in case you are interested in a more extensive list (and references) of reasons for and against each, see below.

Why are p-values undesirable?

• As Darren James notes (and his simulation shows), p-values are largely contingent on the number of observations that you have (see Kirk, 2003)
• As Jon notes, p-values represent the conditional probability of observing data as extreme or more extreme given that the null hypothesis is true. As most researchers would rather have probabilities of the research hypothesis, and/or the null-hypothesis, p-values do not speak to probabilities in which researchers are most interested (i.e., of the null or research hypothesis, see Dienes, 2008)
• Many who use p-values do not understand what they mean/do not mean (Schmidt & Hunter, 1997). Michael Lew’s reference to Gelman and Stern’s (2006) paper further underscores researcher misunderstandings about what one can (or cannot) interpret from p-values. And as a relatively recent story on FiveThirtyEight demonstrates, this continues to be the case.
• p-values are not great at predicting subsequent p-values (Cumming, 2008)
• p-values are often misreported (more often inflating significance), and misreporting is linked to an unwillingness to share data (Bakker & Wicherts, 2011; Nuijten et al., 2016; Wicherts et al., 2011)
• p-values can be (and historically, have been) actively distorted through analytic flexibility, and are therefore untrustworthy (John et al., 2012; Simmons et al., 2011)
• p-values are disproportionately significant, as academic systems appear to reward scientists for statistical significance over scientific accuracy (Fanelli, 2010; Nosek et al., 2012; Rosenthal, 1979)

Why are effect sizes desirable?

Note that I am interpreting your question as referring specifically to standardized effect sizes, as you say they allow researchers to transform their findings “INTO A COMMON metric”.

• As Jon and Darren James indicate, effect sizes indicate the magnitude of an effect, independent of the number of observations (American Psychological Association 2010; Cumming, 2014) as opposed to making dichotomous decisions of whether an effect is there or not there.
• Effect sizes are valuable because they make meta-analyses possible, and meta-analysis drive cumulative knowledge (Borenstein et al., 2009; Chan & Arvey, 2012)
• Effect sizes help to facilitate sample size planning via a priori power analysis, and therefore efficient resource allocation in research (Cohen, 1992)

Why are p-values desirable?

Though they are less frequently espoused, p-values have a number of perks. Some are well-known and longstanding, whereas others are relatively new.

• P-values provide a convenient and familiar index of the strength of evidence against the statistical model null hypothesis.

• When calculated correctly, p-values provide a means of making dichotomous decisions (which are sometimes necessary), and p-values help keep long-run false-positive error rates at an acceptable level (Dienes, 2008; Sakaluk, 2016) [It is not strictly correct to say that P-values are required for dichotomous decisions. They are indeed widely used that way, but Neyman & Pearson used 'critical regions' in the test statistic space for that purpose. See this question and its answers]

• p-values can be used to facilitate continuously efficient sample size planning (not just one-time power-analysis) (Lakens, 2014)
• p-values can be used to facilitate meta-analysis and evaluate evidential value (Simonsohn et al., 2014a; Simonsohn et al., 2014b). See this blogpost for an accessible discussion of how distributions of p-values can be used in this fashion, as well as this CV post for a related discussion.
• p-values can be used forensically to determine whether questionable research practices may have been used, and how replicable results might be (Schimmack, 2014; also see Schönbrodt’s app, 2015)

Why are effect sizes undesirable (or overrated)?

Perhaps the most counter-intuitive position to many; why would reporting standardized effect sizes be undesirable, or at the very least, overrated?

• In some cases, standardized effect sizes aren’t all that they are cracked up to be (e.g., Greenland, Schlesselman, & Criqui, 1986). Baguely (2009), in particular, has a nice description of some of the reasons why raw/unstandardized effect sizes may be more desirable.
• Despite their utility for a priori power analysis, effect sizes are not actually used reliably to facilitate efficient sample-size planning (Maxwell, 2004)
• Even when effect sizes are used in sample size planning, because they are inflated via publication bias (Rosenthal, 1979) published effect sizes are of questionable utility for reliable sample-size planning (Simonsohn, 2013)
• Effect size estimates can be—and have been—systemically miscalculated in statistical software (Levine & Hullet, 2002)
• Effect sizes are mistakenly extracted (and probably misreported) which undermines the credibility of meta-analyses (Gøtzsche et al., 2007)
• Lastly, correcting for publication bias in effect sizes remains ineffective (see Carter et al., 2017), which, if you believe publication bias exists, renders meta-analyses less impactful.

Summary

Echoing the point made by Michael Lew, p-values and effect sizes are but two pieces of statistical evidence; there are others worth considering too. But like p-values and effect sizes, other metrics of evidential value have shared and unique problems too. Researchers commonly misapply and misinterpret confidence intervals (e.g., Hoekstra et al., 2014; Morey et al., 2016), for example, and the outcome of Bayesian analyses can distorted by researchers, just like when using p-values (e.g., Simonsohn, 2014).

All metrics of evidence have won and all must have prizes.

References

American Psychological Association. (2010). Publication manual of the American Psychological Association (6th edition). Washington, DC: American Psychological Association.

Baguley, T. (2009). Standardized or simple effect size: What should be reported?. British Journal of Psychology, 100(3), 603-617.

Bakker, M., & Wicherts, J. M. (2011). The (mis) reporting of statistical results in psychology journals. Behavior research methods, 43(3), 666-678.

Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Sussex, UK: John Wiley & Sons, Ltd.

Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2017, August 12). Correcting for bias in psychology: A comparison of meta-analytic methods. Retrieved from osf.io/preprints/psyarxiv/9h3nu

Chan, M. E., & Arvey, R. D. (2012). Meta-analysis and the development of knowledge. Perspectives on Psychological Science, 7(1), 79-92.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. 

Cumming, G. (2008). Replication and p intervals: p values pre- dict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science, 3, 286– 300.

Dienes, D. (2008). Understanding psychology as a science: An introduction to scientific and statistical inference. New York, NY: Palgrave MacMillan.

Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PloS one, 5(4), e10068.

Gelman, A., & Stern, H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4), 328-331.

Gøtzsche, P. C., Hróbjartsson, A., Marić, K., & Tendal, B. (2007). Data extraction errors in meta-analyses that use standardized mean differences. JAMA, 298(4), 430-437.

Greenland, S., Schlesselman, J. J., & Criqui, M. H. (1986). The fallacy of employing standardized regression coefficients and correlations as measures of effect. American Journal of Epidemiology, 123(2), 203-208.

Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E. J. (2014). Robust misinterpretation of confidence intervals. Psychonomic bulletin & review, 21(5), 1157-1164.

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. PsychologicalSscience, 23(5), 524-532.

Kirk, R. E. (2003). The importance of effect magnitude. In S. F. Davis (Ed.), Handbook of research methods in experimental psychology (pp. 83–105). Malden, MA: Blackwell.

Lakens, D. (2014). Performing high‐powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701-710.

Levine, T. R., & Hullett, C. R. (2002). Eta squared, partial eta squared, and misreporting of effect size in communication research. Human Communication Research, 28(4), 612-625.

Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychological methods, 9(2), 147.

Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E. J. (2016). The fallacy of placing confidence in confidence intervals. Psychonomic bulletin & review, 23(1), 103-123.

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615-631.

Nuijten, M. B., Hartgerink, C. H., van Assen, M. A., Epskamp, S., & Wicherts, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior research methods, 48(4), 1205-1226.

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641.

Sakaluk, J. K. (2016). Exploring small, confirming big: An alternative system to the new statistics for advancing cumulative and replicable psychological research. Journal of Experimental Social Psychology, 66, 47-54.

Schimmack, U. (2014). Quantifying Statistical Research Integrity: The Replicability-Index. Retrieved from http://www.r-index.org 

Schmidt, F. L., & Hunter, J. E. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 37–64). Mahwah, NJ: Erlbaum.

Schönbrodt, F. D. (2015). p-checker: One-for-all p-value analyzer. Retrieved from http://shinyapps.org/apps/p-checker/. 

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366.

Simonsohn, U. (2013). The folly of powering replications based on observed effect size. Retreived from http://datacolada.org/4

Simonsohn, U. (2014). Posterior-hacking. Retrieved from http://datacolada.org/13.

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534-547.

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science, 9(6), 666-681.

Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PloS one, 6(11), e26828.

Answer 5

From the perspective of an Epidemiologist, on why I prefer effect sizes over p-values (though as some people have noted, it's something of a false dichotomy):

1. The effect size tells me what I actually want, the p-value just tells me if it's distinguishable from null. A relative risk of 1.0001, 1.5, 5, and 50 might all have the same p-value associated with them, but mean vastly different things in terms of what we might need to do at a population level.
2. Relying on a p-value reinforces the notion that significance-based hypothesis testing is the end-all, be-all of evidence. Consider the following two statements: "Doctors smiling at patients was not significantly associated with an adverse outcome during their hospital stay." vs. "Patients who had their doctor smile at them were 50% less likely to have an adverse outcome (p = 0.086)." Would you still, maybe, given it has absolutely no cost, consider suggesting doctors smile at their patients?
3. I work with a lot of stochastic simulation models, wherein sample size is a function of computing power and patience, and p-values are essentially meaningless. I have managed to get p < 0.05 results for things that have absolutely no clinical or public health relevance.