Can I interpret logistic regression coefficients and their p-values even if model performance is bad?

Question

I trained a logistic regression model with two binary variables $x_1$ and $x_2$ where my target binary variable is $y$:

$$\log\left(\frac{p}{1-p}\right) =\beta_0+\beta_1x_1+\beta_2x_2$$ Looking at p-values, we notice that $x_1$ has statistical significance (p-value < 0.05).
However, the model performance is terrible (pseudo-R-squared = 0.004552).
So I would like to know if it is reasonable to conclude that there is evidence that $x_1$ is related to outcome $y$ or not.

Note: I am using the statsmodel Python library

Answer 1

I wanted to add a visual example reproducing results similar to the OP. (I usually understand things when I write the code for them).

Here we have 2000 datapoints. The true relationship between x and y is $y = 0.1x$, but the data is very noisy:

set.seed(1234)
N <- 2000
x <- seq(1, 5, length.out=N)
b <- 0.1
y <- b * x + rnorm(n=N, mean=0, sd=1)
plot(x, y, pch=19, cex=0.5)

A large amount of variation remains unexplained, $R^{2} \approx 0.02$. However, the linear model correctly gives a good estimate of the regression coefficient (~0.12) which is convincingly different from zero ($p \approx 10^{-9}$):

fit <- lm(y ~ x)
summary(fit)
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.05891    0.06156  -0.957    0.339    
x            0.11762    0.01915   6.143 9.76e-10 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.9893 on 1998 degrees of freedom
Multiple R-squared:  0.01854,   Adjusted R-squared:  0.01804 
F-statistic: 37.73 on 1 and 1998 DF,  p-value: 9.761e-10

So it's ok to interpret this model since the assumptions of the linear regression are satisfied. However, what you want to take away from it is a different question that depends on your context.

Answer 2

1893 observations is a lot. All kinds of hypothesis tests will yield statistical significance even for tiny effects if the sample size is large (we have many threads on this, here is a similar question on ANOVA I recently had).

Thus, it looks like there is signal in your data, it's just very weak. Some effects in the world are like this. Your model might indeed be the best you can do, or perhaps you can improve it (but beware of p-hacking!).

You can definitely interpret this, just keep the caveat about the weakness of the effect in mind.

Answer 3

It is routine in many areas to do inference with models that have low performance. I cannot think of any examples with $R^2$ values as low as the pseudo $R^2$ you have, but I know I've read papers in high-end journals that do inference on models with linear regression $R^2$ values around $4\%$, maybe even lower.

A cause for concern is if you've missed something in the modeling. For instance, it could be that there is a generally upward trend, but only because the data form a checkmark shape. In that case, your model is misleading. For low values of your feature, increase the value actually decreases the outcome, and you might find yourself frustrated if you apply your bogus model to such values only to find the outcome decreasing when the regression says it should increase.

However, a good regression modeling strategy should help you avoid such a situation, so if you have been thorough in your modeling, the low performance need not be interpreted as the regression screaming, "I'm worthless," at you.

Answer 4

If you are a trader and you can get an pseuo-R² of 0.001 in predicting future financial transactions, you're the richest man in the world. If you are a CPU engineer and your design gets a pseudo-R² of 0.999 at telling if a bit is 0 or 1, you have a useless piece of silicon.

This statement is a copy from the question: Determine how good an AUC is (Area under the Curve of ROC)

Whether or not a particular value for R² is actually considered good performance, that depends on the problem.

---

Also note that R² is not a goodness of fit measure. So the value doesn't tell directly whether your model is a good fit or not. Instead, it tells how large the noise/randomness is relative to the deterministic part. (It's background knowledge about the problem that tells whether a particular ratio, a particular R² a particular performance also means whether or not a model is a good fit or not)

R² and pseudo-R² can be, due to it's computation never be high, even for the very best model. It is not a measure of goodness of fit that determines whether or not we are close to the true model of the distribution.

For instance, there can simply be a lot of noise. Cohen's pseudo-R² is a ratio of deviance $(D_{null}-D_{fitted})/D_{null}$. This value of $D_{fitted}$ does not need to approach zero when the fitted model approaches the perfect model. For binary distributed variables there will always be randomness and we are predicting the population parameters not the binary outcomes.

Instead, goodness of fit can be tested with, for instance, a chi-squared test or G-test (but these require multiple measurements at the same conditions, ie same regressor values).

Answer 5

I would tentatively say, "yes". Remember that pseudo-$R^2$ is an analogue to $R^2$, which in OLS is variance explained. Pseudo-$R^2$ is not that, but it's roughly similar. So your low pseudo-$R^2$ could mean you're missing an explanatory variable, a random effect etc. (many reasons possible). In other words, it's a not very good model for describing the data. It would thus likely be a poor predictive model. But, if you're just doing a hypothesis test, then even though you have a lot of extra variance that you can't explain, we're still finding a significant difference between the two x1 groups.

Again, I'm just saying this tentatively. If you want to know whether you should use this model, it also depends on your study aims and your analysis methods. Do you need an predictive model, or are you just looking to see if x1 has an effect? Do you need to control for multiple comparisons? etc.