Should I remove the intercept when I have one dummy variable that covers all the categories in a categorical variable?

Question

I have a categorical variable that has $4$ categories, and I have two dummy variables, $x_1$ and $x_2$, that cover this categorical variable. The $x_1$ variable has values of only $1$ without any zeroes $[1,1,1,1,1...1]$, while $x_2$ has values of both $0$ and $1$ $[0,1,1,0,0,0...0]$. When I use multiple linear regression model with these two dummy variables as regressors, I get missing values at the output of summary(model) function in R (NA).

My questions are:

1. Should I keep the regressor or remove it from the model?
2. What will happen to the interpretations of the parameter estimates when I remove the intercept?
3. Is there a situation where can we freely remove the intercept without affecting the interpretation of the parameter estimates?
4. Why does R-squared increase when I remove the intercept?

This is the model with the intercept:

Call:
lm(formula = output ~ X1 + X2, data = data)
Residuals:
    Min      1Q  Median      3Q     Max 
-3.3091 -1.2591 -0.0091  1.2415  3.6909
Coefficients: (1 not defined because of singularities)
            Estimate Std. Error t value Pr(>|t|)

(Intercept)  10.3083     0.5329  19.345   <2e-16 ***
X1                NA         NA      NA       NA

X2           -0.3992     0.6012  -0.664    0.509

Signif. codes:  0 ‘*’ 0.001 ‘’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 1.846 on 54 degrees of freedom
Multiple R-squared:  0.008101,  Adjusted R-squared:  -0.01027 
F-statistic: 0.441 on 1 and 54 DF,  p-value: 0.5094

This is the model without the intercept:

Call:
lm(formula = output ~ X1 + X2 - 1, data = data)
Residuals:
    Min      1Q  Median      3Q     Max 
-3.3091 -1.2591 -0.0091  1.2415  3.6909
Coefficients:
   Estimate Std. Error t value Pr(>|t|)

X1  10.3083     0.5329  19.345   <2e-16 ***
X2  -0.3992     0.6012  -0.664    0.509

Signif. codes:  0 ‘*’ 0.001 ‘’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 1.846 on 54 degrees of freedom
Multiple R-squared:  0.9682,    Adjusted R-squared:  0.967 
F-statistic: 821.1 on 2 and 54 DF,  p-value: < 2.2e-16

This is the model without the regressor X1:

Call:
lm(formula = output ~ X2, data = data)
Residuals:
    Min      1Q  Median      3Q     Max 
-3.3091 -1.2591 -0.0091  1.2415  3.6909
Coefficients:
            Estimate Std. Error t value Pr(>|t|)

(Intercept)  10.3083     0.5329  19.345   <2e-16 ***
X2           -0.3992     0.6012  -0.664    0.509

Signif. codes:  0 ‘*’ 0.001 ‘’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 1.846 on 54 degrees of freedom
Multiple R-squared:  0.008101,  Adjusted R-squared:  -0.01027 
F-statistic: 0.441 on 1 and 54 DF,  p-value: 0.5094

Answer 1

I think the issue is quite obvious from what I can see. Your $x_1$ variable here only takes on one value, therefore it is simply a linear constant aka it doesn't contribute any new information to the model (it also appears there are problems with $x_2$ but I'll remark on that later). To your questions:

Should I keep the regressor or remove it from the model?

Remove it. It adds nothing to your model.

What will happen to the interpretations of the parameter estimates when I remove the intercept?

With respect to a regression that only includes categorical predictors, there is nothing wrong with removing the intercept. It simply makes the intercept/slopes more interpretable (they just represent the contrasts between levels of each factor).

Is there a situation where can we freely remove the intercept without affecting the interpretation of the parameter estimates?

In this case it isn't an issue. It can also be okay to do when $x=0$ and $y=0$ is a real possibility (Eisenhauer, 2003; Hahn, 1977), but generally speaking it is better to leave the intercept in.

Why does R-squared increase when I remove the intercept?

This is normal. Suppressing the intercept means that you are just predicting the model based off mean adjustments between groups, which without including any slope terms, means that you are capturing most/all of the variation in the DV as apportioned by each group. But again I think that $x_1$ here doesn't add anything to your model so you can simply remove it. Notice that Model 1 and 3 have no change despite actively removing the variable.

One final note...your $x_2$ variable doesn't seem to contribute much either though. This is why your adjusted $R^2$ is also negative (see a visualization of this issue here with some brief explanation in Chicco et al., 2021). Your model basically has no predictive value. As an example of a dummy coded (binary) variable fit with no relationship to the outcome:

#### Simualate Data ####
set.seed(123)
n <- 200
x <- rbinom(n,1,.5)
y <- rnorm(n)
Fit and Summarize
fit <- lm(y ~ x+0)
summary(fit)
Plot
plot(x,y,
     main="Negative R2 Fit")
abline(fit,
       col="red",
       lwd=2)

We get a negative $R^2$, and our regression line fit is close to horizontal (note that using an intercept or suppressing it here doesn't make a difference, I just show a similar case here to yours).

Edit

It occurs to me now that you also haven't coded your variable as a factor. The issue is it's not coding your 0 and 1 values as actual dummy codes, though I'm not sure if that actually fixes your problem (for example y ~ factor(x)). Suppressing categorical regression intercepts doesn't affect the fitting and only changes the interpretation (they simply represent the contrast in means between groups). For an example of what I mean, here I fit a similar set of simulated data with 4 levels this time and coded as numeric and as factor:

#### Set Seed ####
set.seed(123)
n <- 200
x2 <- rbinom(n,3,.5)
y2 <- rnorm(n)
Fit and Summarize
fit.num <- lm(y2 ~ x2 - 1)
fit.cat <- lm(y2 ~ factor(x2)-1)
summary(fit.num)
summary(fit.cat)

You can see one estimates the slope as a numeric variable, whereas the other actually contrasts by group (dummy codes). Both are still poorly fit.

References

• Chicco, D., Warrens, M. J., & Jurman, G. (2021). The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation. PeerJ Computer Science, 7, e623. https://doi.org/10.7717/peerj-cs.623
• Eisenhauer, J. G. (2003). Regression through the origin. Teaching Statistics, 25(3), 76–80. https://doi.org/10.1111/1467-9639.00136
• Hahn, G. J. (1977). Fitting regression models with no intercept term. Journal of Quality Technology, 9(2), 56–61. https://doi.org/10.1080/00224065.1977.11980770