8

I'm currently reading a paper which utilises a multiple regression and reports the adjusted $R^2$ with a p-value, and I'm wondering what this p-value refers to. Can you calculate p-values for adjusted $R^2$? It separately reports the p-values of the t-statistics for each exogenous variable so it can't be t-statistics.

Table

Richard Hardy
  • 67,272
mrepic1123
  • 489
  • 6
  • @StephanKolassa, you should consider to make this comment as the answer as it answers the question very well. The only other possible answer I can think of is bootstrapped estimates. – POC Mar 21 '24 at 13:40
  • @StephanKolassa thank you for your answer! It does say in the paper that it refers to post-hoc power. Could I ask why this is so problematic? – mrepic1123 Mar 21 '24 at 13:43
  • 1
    Could you post the link to the paper? Otherwise it is only possible to guess – Firebug Mar 21 '24 at 13:51
  • 1
    I reposted my comment as an answer. It's always hard to prove a negative... and I toned down my comments on post hoc power from "deeply problematic" to "problematic", because in the end, it's not misleading, it's just vacuous. – Stephan Kolassa Mar 21 '24 at 13:53
  • The R squared, under the null, has a beta distribution with a certain expectation, which, if I recall correctly, equals the difference between the two R-squared. – Michael M Mar 21 '24 at 14:04
  • 2
    Because a p-value for an adjusted $R^2$ is nonsense (an adjusted $R^2$ is not a property of the distributional model), I suspect the intention is that the adjusted $R^2$ is a test statistic being used to test the hypothesis that the true $R^2$ is nonzero. – whuber Mar 21 '24 at 16:16
  • @MichaelM, the null distribution of $R^2$ is discussed here: https://stats.stackexchange.com/questions/130069/what-is-the-distribution-of-r2-in-linear-regression-under-the-null-hypothesis – Christoph Hanck Mar 22 '24 at 10:14

4 Answers4

11

I am not aware of much theoretic understanding of the distribution of $R^2$ or adjusted $R^2$ under the null hypothesis, whether asymptotic or not, which would be required to calculate a $p$ value. As POC notes, it might be possible to obtain a $p$ value through bootstrapping. Alternatively, one could run a permutation test. However, if the authors of the paper did something like this, one would expect that they noted it in the text.

What is indeed common is reporting the $p$ value of an $F$ test for the ANOVA between a given model and an intercept-only model. It may well be that the authors' software reported the $F$ statistic (with its degrees of freedom), the $R^2$ and the $p$ value for the $F$ test, and the authors only reported $R^2$ and $p$.

What is a little more concerning to me is the "power" column, which I fear refers to post hoc power, which is problematic. It simulates understanding we do not have, because if a significant effect was found, then the study was by definition sufficiently powerful to detect this effect as significant. Post hoc power reasons in a circle. The proper place for power calculations is in planning a study and its sample size, using effect sizes "we would be sorry to miss". On post hoc power, I like to recommend Hoenig & Heisey (2001).

Stephan Kolassa
  • 123,354
  • 1
    You wouldn’t consider the overall F-test to be some kind of test of $R^2$, if the residual variance of statistically significantly lower than the total variance? – Dave Mar 21 '24 at 14:05
  • 1
    @Dave: yes, $R^2$ and $F$ of course hang together very closely, so one could consider the $F$ test in an ANOVA a kind of a test for $R^2$. – Stephan Kolassa Mar 21 '24 at 14:07
  • 3
    The distribution of $R^2$ is so closely related to the distribution of the correlation coefficient that I'm sure you know how to find it under the null hypothesis $R^2=0$ (making the implicit Normality assumptions, of course). – whuber Mar 21 '24 at 14:13
  • @whuber, how should we interpret a null like $R^2=0$ when $R^2$ is a sample quantity. Population $R^2$? Also, we would be testing on the boundary of a parameter space, no, which often is not so trivial? – Christoph Hanck Mar 22 '24 at 10:12
  • 1
    @Dave, this one discusses $F$ and $R^2$: https://stats.stackexchange.com/questions/597947/relationship-between-f-fisher-and-r2/598006#598006 – Christoph Hanck Mar 22 '24 at 10:15
  • 1
    @Christoph $R^2$ is indeed a property of the underlying model: it is not merely a sample quantity. The adjusted $R^2$ clearly is a sample quantity (because it depends on sample size). For instance, in a Bivariate normal model of ordinary regression with correlation parameter $\rho,$ the population $R^2$ equals $\rho^2.$ The point about boundaries is good, but is obviated when you instead test the equivalent hypothesis $\rho=0$ against $\rho\ne 0:$ there's no longer any boundary! – whuber Mar 22 '24 at 14:07
  • 1
    Given whuber's first comment and Sextus's answer, I find the first sentence surprising. While it is phrased as your knowledge rather than some consensus, do you still stick to it? – Richard Hardy Mar 23 '24 at 17:51
4

$R^2$ and adjusted $R^2$ are computed based on a decomposition of the sum of squares, $SS_{total} = SS_{model}+SS_{residual}$, and can be related to F-statistics in ANOVA which is based on the same sum of squares values.

For simple linear regression an example is given here: Is there a relation between the p-values of coefficients and the $R^2$ in an OLS regression?

It separately reports the p-values of the t-statistics for each exogenous variable so it can't be t-statistics.

The models contain several variables for which you can seperately compute p-values, by computing t-values, or computing ANOVA by comparing with a model that drops a single factor (unless the factor being a categorical variable, this should give the same result).

The p-value for the entire model is an ANOVA comparison between a null model and the entire model. In such a case $R^2$ can be related to the F statistic and it's p-value in a similar way as the simple linear regression linked above.

(In the case of the 'log-dimensionless jerk' the model is actually just a simple linear regression and the 'p-value of the parameter', is the same as the 'p-value of the entire model'.)

Example computation in R

### generate some data
x1 = c(1,5,3,7,2,5,4,9,0)
x2 = c(1,1,1,1,0,0,0,0,0)
set.seed(1)
noise = rnorm(9)
### simulate a measurement 
y = 2 + x1 + x2 + noise

compute two models


mod_full = lm(y ~ 1 + x1 + x2) ## intercept plus both variables
mod_null = lm(y ~ 1) ## only intercept


compare the full model with an intercept only model


anova(mod_full,mod_null)


p-value = 0.0001191


provide p-values of individual variables


summary(mod_full)


x1 p-value = 3.99e-05


x2 p-value = 0.2013

The p-values for the variables are evaluated with a t-test and result in

Call:
lm(formula = y ~ 1 + x1 + x2)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)

(Intercept)   1.8019     0.5646   3.191   0.0188 *

x1            1.1151     0.1045  10.672 3.99e-05 ***
x2            0.8171     0.5694   1.435   0.2013

The p-value for the model is evaluated with an ANOVA test and results in

Model 1: y ~ 1 + x1 + x2
Model 2: y ~ 1
  Res.Df    RSS Df Sum of Sq      F    Pr(>F)    
1      6  4.323                                  
2      8 87.867 -2   -83.544 57.976 0.0001191
Sextus Empiricus
  • 77,915
  • Digressing, instead of computing t-tests for the individual variables one can also do an ANOVA test with type III sums car::Anova(mod_full, type = "III"). – Sextus Empiricus Mar 22 '24 at 08:57
2

The adjusted coefficient of determination is a monotonically increasing function of the F-statistic, so both statistics give the same evidentiary ordering in a goodness-of-fit test. Specifically, it can be shown that:

$$\begin{align} F &= \frac{df_\text{Res}}{df_\text{Reg}} \cdot \frac{R^2}{1-R^2} \\[6pt] &= \frac{df_\text{Res}}{df_\text{Reg}} \cdot \frac{(n-1) - (n-p-1)(1-R_\text{Adj}^2)}{(n-p-1)(1-R_\text{Adj}^2)} \\[6pt] &= \frac{df_\text{Res}}{df_\text{Reg}} \cdot \frac{p + (n-p-1)R_\text{Adj}^2}{(n-p-1) (1-R_\text{Adj}^2)}, \\[6pt] \end{align}$$

which gives:

$$R_\text{Adj}^2 = \frac{(n-p-1) df_\text{Reg} \cdot F - p \cdot df_\text{Res}}{(n-p-1) df_\text{Reg} \cdot F + (n-p-1) df_\text{Res}}.$$

Since this is a monotonically increasing function of $F$, the goodness-of-fit test for the $R_\text{Adj}^2$ statistic ought to be identical to the standard F-test.

Ben
  • 124,856
-1

Assuming the authors used R (I did not read the paper!), they likely used the cor.test function with default parameters to obtain those numbers. Here is the relevant section from the online help:

If ‘method’ is ‘"pearson"’, the test statistic is based on Pearson's product moment correlation coefficient ‘cor(x, y)’ and follows a t distribution with ‘length(x)-2’ degrees of freedom if the samples follow independent normal distributions. If there are at least 4 complete pairs of observation, an asymptotic confidence interval is given based on Fisher's Z transform.

mschilli
  • 119