4

I'm testing if economic growth before an election is correlated with vote percentage in elections. So I have one independent variable in my model.

The problem is: my independent variable is significant but my $R^2$ is very low ($0.0045$) and the $F$ value is significant too. I am testing the correlation between these two variables and not going to predict. Should I worry about the $R^2$ value, or can I just calculate correlation without a regression model?

Shawn Hemelstrand
  • 13,543
Sina Alvandi
  • 43

2 Answers2

5

There are plenty of cases where a low $p$ value can be obtained with a very low effect size, here the $R^2$. I show a simulated example here, where even miniscule effect sizes can drive statistically significant results with enough data. From that answer, I show how a statistically significant result is driven by a very weak effect size simply by having thousands of observations.

enter image description here

A deeper discussion on $p$ values can be found here if you are interested in the history and interpretation of them (you can probably skip to the part about the ASA statement on the "definition" of $p$-values). I would like to note that for your example, it is not wise to change statistical testing purely because of the results of a significance test.

Shawn Hemelstrand
  • 13,543
  • I couldn't get my answer. In this case according to p value for my independent variable (growthe one year before election) I can see economic growth one year before election has significant effect on election vote percentage. But R^2 is very low. I think it suggests that there are correlation between these variables (this is what I want to say in my paper) but growth can not explain vote percentage because R is low (which is not important for me because I don't want to predict vote by growth). Is it true? – Sina Alvandi Mar 10 '24 at 07:39
  • 1
    I think this is a fundamental misunderstanding of how $p$ values work. A $p$ value does not say there is an effect, only the probability of observing a given point estimate as extreme or more if the null hypothesis is true. That says nothing about the actual magnitude of the effect, as noted in my answer. What you say doesn't make sense about suggesting there is a correlation given that the $R^2$ for a simple regression is the squared correlation between the two variables. – Shawn Hemelstrand Mar 10 '24 at 07:55
2

You can't ignore the R-squared value. R-squared exists to show you how well your model can explain a phenomenon. E.g. how well do changes in your input/independent variable values predict the actual outcome/dependent variable values. Statistical Significance illustrates the degree of confidence that your model is depicting actual relationships between variables vs random chance.

In your particular case, the combination of the two results is basically telling you that the relationship between those two variables is more than likely real but not useful for explanatory purposes. From this I would take away that you are measuring a real effect but that the relationship is too weak to be of any practical use.

It is reasonably likely that you have omitted variables that need to be included in order to properly model the phenomon being studied. (Note: adding variables simply to improve R-score and P values can present an ethical issue if you're simply adding or removing variable to optimize your results. But if you have legitimate theoretical justification for adding variables, then that isn't really an issue in and of itself (though this will present other potential issues related to multiple regression that are seemingly beyond the scope of this question.)

Bottom line: in order for a model to be a useful explanatory tool, it needs to be significant and have a high r-squared value.

P.S. In multiple regression, the F-test p-value tests the significance of the entire model while the p-values for the coefficients test the significance of that particular variable. and yes, it is possible to have a significant model with one or more insignificant variables. I had that problem with a recent analysis because it turned out that 3 of my variables were mediators for my fourth variable and I had to account for that and test for mediation.

Sam Levi
  • 69
  • 3
    As I can't reply to other comments yet, I will add my thoughts here. But as Shawn heavily implied above, exceptionally large data sets can generate results that have statistical significance but don't have any meaningful practical significance. I would argue that this is still a meaningful result as it tends to confirm the fact that your chosen IV isn't a valid predictor for explaining the dependent variable. – Sam Levi Mar 09 '24 at 07:13
  • 1
    I agree with the sentiments from Sam. Just because an effect is weak doesn't mean it is not important. Null or near null effects are important for the field, and they are especially interesting when we expect otherwise. – Shawn Hemelstrand Mar 09 '24 at 08:51