Removing variables from a linear regression improves $R^2_{adj.}$

Question

I am working on a linear regression model.

The complete model with 11 variables in total has a quite low adjusted R-squared ($R^2_{adj.}$) of 0.11.
4 variables have a significant influence on the DV.
Alternative model: using stepwise model selection based on the AIC, a model with 7 variables turns out to have the best AIC. Further, $R^2_{adj.}$ improved comparatively much and equals 0.16 in the alternative model.
in the alternative model, the same 4 variables have an influence on the DV and a further, prior non-significant variable turns to a significant one.

This finding is somewhat not intuitive. Based on what I know so far about the $R^2_{adj.}$ measure is that it is a measure which indicates how much of variance is explained by the model.
It usually increases when adding more variables as it is likely that more variables explain more variance.

Conceptually, the complete model is theory deduced.

Does anyone have an idea, why the $R^2_{adj.}$ increases when removing variables?

After hours of research, I still did not find an appropriate explanation for this result.

I am looking at the adjusted R2. the not adjusted R2 remains almost constant for both models (0.26 for the complete model and 0.25 for the alternative model) — urmelf, Jul 23 '21 at 10:08
So the actual $R^2$ does increase with the extra variable, in line with your intuition, but not by enough to offset the impact of the adjustment for the number of variables. This is not unusual. — Henry, Jul 23 '21 at 10:27
Hi Henry, thank you for your answer. So should I rather use the alternative model? — urmelf, Jul 23 '21 at 10:29
See also "Justification for and optimality of $R^2_{adj.}$ as a model selection criterion". — Richard Hardy, Jul 23 '21 at 14:25

score 3 · Accepted Answer · answered Jul 23 '21 at 12:19

You are using $R^{2}_{adj}$ incorrectly. It is intended to debias $R^2$ for a single pre-specified model. When used in the context of multiple models the degrees of freedom that needs to be inserted into the formula is the maximum number of parameters entertained across models. Then your problem also vanishes.

score 1 · Answer 2 · answered Jul 23 '21 at 11:12

1

It is the $R^2$ which indicates how much of variance is explained by the model. You confuse $R^2$ and $R^2_{adj}$, you should read this to fully understand the difference between both of them : https://www.statisticshowto.com/probability-and-statistics/statistics-definitions/adjusted-r2/

$R^2_{adj}$ can be use as a criterion for step-wise method, such as AIC or BIC : it penalizes the adding of variables.

answered Jul 23 '21 at 11:12

Romain

13

2

Regarding your last paragraph, this might be a bad idea: see "Justification for and optimality of $R^2_{adj.}$ as a model selection criterion". – Richard Hardy Jul 23 '21 at 14:25
1

Totally agree, AIC and BIC are way better for model selection, my last paragraph was only illustrative for the penalty in adding new variable. @RichardHardy : As an expert in time series, could you please take a loot at a question I've posted ? https://stats.stackexchange.com/questions/535658/include-exogenous-variable-in-both-level-and-first-difference – Romain Jul 23 '21 at 14:56

Removing variables from a linear regression improves $R^2_{adj.}$

2 Answers2