Low R-squared for binary logistic regression model but all variables are significant

Question

I am currently testing a binary logistic regression model (N=2000), examining the relationship between several independent variables (such as substance use -categorial-, gender-categorial-, self-control-likert-) and a binary/categorical dependent variable related to delinquency (no crime-at least one crime). However, I am facing a challenge with the Nagelkerke R-squared, which appears to be lower than expected... really really low. Despite all the independent variables showing statistical significance, the overall explanatory power of the model seems inadequate.

I used five independent variables in my analysis and would appreciate insights into why the Nagelkerke R-squared is low despite the significance of the individual predictors. Additionally, I am curious if there are alternative ways to improve the R-squared or if there are other indicators that could provide a more comprehensive assessment of the model's performance.

Any guidance or suggestions would be greatly appreciated. Thank you!

Answer 1

You have a fairly large sample size of $2000$, meaning that you have the power to detect even small effects. What you're seeing can be interpreted as strong evidence of a weak relationship. I have likened this to the Princess and the Pea fairy tale where the princess has such sensitive skin that she can detect even the tiny pea under the mattress [1, 2 3].

It is typical to see this kind of behavior. For instance, in the paper I mention in this answer of mine, the performance was very modest despite statistical significance of the coefficients.

For instance, see the simulation below where a low Nagelkerke $R^2$ less than $0.03365$ is achieved despite each regression coefficient being significant at the $0.05$-level.

library(MLmetrics)
set.seed(2024)
N <- 2000
p <- 5
X <- matrix(runif(N*p, 0, 1), N, p)
B <- rep(0.7, p)
z <- X %*% B - 0.7
p <- 1/(1 + exp(-z))
y <- rbinom(N, 1, p)
model <- glm(y ~ X, family = binomial)
null  <- glm(y ~ 1, family = binomial)
Nagelkerke R^2

nag_r2 <- function(L_model, L_null, n){
numerator <- 1 - (L_null / L_model)^(2/n)
  denominator <- 1 - (L_null)^(2/n)
  return(numerator/denominator)
}
Exponentiate the log-likelihood to get the likelihood

L_model <- exp(MLmetrics::LogLoss(predict(model, type = "response"), y))
L_null  <- exp(MLmetrics::LogLoss(predict(null,  type = "response"), y))
summary(model)
nag_r2(L_model, L_null, N)

Answer 2

p values are partly an effect of sample size. With a large sample size (and 2000 is pretty big) small effects can be significant.

For measures of fit for logistic regression see Allison, Measures of fit for Logistic Regression, presented and published at SUGI (SAS Users group) and available on the web. While Allison uses SAS a bit here, that's really not important. You should read the whole paper, but, briefly, he recommends

1. Standardized Pearson Test
2. Unweighted sum of squares
3. Information matrix test
4. Stukel's test.

He shows how to implement these in SAS, this might be used by someone to implement them in R or Python. Someone may have already done this.

Answer 3

The frequency distribution of Y was not provided. In the most information-rich case of 1000 in each of two groups, the effective sample size is $2000 \times \frac{1}{2} \times \frac{1}{2} \times 3 = 1500$. If the distribution is imbalanced, the effective sample size can be much smaller. For example if a fraction of 0.1 of the observations have Y=1, the effective sample size is only 540. The Cox-Snell and Nagelkerke pseudo $R^2$ using the apparent sample size and not the effective sample size in their calculations. As done with the R rms package lrm function, pseudo and adjusted pseudo $R^2$ calculations are done using the effective sample size. This is described here.