If we have $k$ categories of a categorical variable, why do we need $k-1$ dummy variables? For example, if there are 8 categories, why don't we code them as
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1?
Only 3 dummy variables would be needed as opposed to the commonly used $8-1 = 7$ dummy variables.
For one, the predictor variables would not be orthogonal. Generally, linear regressions work better (i.e., have tighter error estimates) when predictors are close to orthogonal (which can be quantified using the condition number of the Gram matrix $X^tX$). But the bigger issue with this proposed coding scheme is that it introduces similarity structure to the features that is completely arbitrary. For example the first row and the second row are less similar than the first row and the third row (as measured by, e.g., dot product similarity), but in a categorical variable the labeling of the values (or what is the same thing, the ordering of the rows of the encoding matrix) has no inherent meaning. So combining these two considerations, the only way that really makes sense to code a categorical variable with $K$ values is to choose $k$ orthogonal vectors, which requires a space of dimension $>=K$ (or really, $>=K-1$ since you can absorb one of them into the intercept). The typical coding using one-hotes is a particularly simple way to accomplish this, although in principle you could also use any set of $K$ orthogonal vectors.
Think about how linear models work. The predicted value is $$ \hat{y}_i = \sum_j \beta_j x_{ji} $$ With dummy variables (aka "one-hot encoding"), for each $i$, only one of the $\beta$, the one corresponding to the level of the categorical variable, is added to the sum. Thus, each level gets its own coefficient that defines the outcome's response when the categorical variable takes on that value$^\star$.
Now, consider what happens with your proposed encoding. Levels 1 and 2 each get their own response, but Level 3 is constrained to be the sum of the responses for levels 1 and 2. There is no reason to suppose that this should be the case, so you shouldn't build that constraint into the model. Having a separate dummy variable for each level allows each level to have its own response that does not depend in any way on the other levels.
$^\star$ Well, except for the one special level that gets its response rolled into the intercept term, but you shouldn't do that. If there are $k$ levels, use $k$ dummy variables and no intercept instead of $k-1$ plus an intercept. For example, if you're doing the model in R, use $y\sim x+0$ instead of $y\sim x$.
y~x+0. It seems you dislike using $k-1$ dummies b/c you have to interpret coefs as contrasts... but isn't that better than having no clear interpretation to the coefs?
– civilstat
Oct 30 '22 at 14:44
When you do the standard encoding, you are using $0$ and $1$ as “off” and “on”. When you give a $1$, you turn on the effect of being in the corresponding category over (or under, or maybe neither) the baseline category that is subsumed by the intercept.
With your encoding, that interpretation is lost. For instance, what is the difference between $001$ and $101$ in your coding scheme? Further, what does it mean if $x_1=1?$
Arguably (much) worse, the model performance depends on how you do the coding.
set.seed(2022)
N <- 100
categories <- rep(c("a", "b", "c", "d", "e", "f", "g", "h"), rep(N, 8))
y <- seq(1, length(categories), 1) #rnorm(length(categories))
x1_1 <- x2_1 <- x3_1 <- x1_2 <- x2_2 <- x3_2 <- rep(NA, length(y))
for (i in 1:length(y)){
if (categories[i] == "a"){
x1_1[i] <- 0
x2_1[i] <- 0
x3_1[i] <- 1
x1_2[i] <- 0
x2_2[i] <- 0
x3_2[i] <- 0
}
if (categories[i] == "b"){
x1_1[i] <- 0
x2_1[i] <- 1
x3_1[i] <- 0
x1_2[i] <- 0
x2_2[i] <- 0
x3_2[i] <- 1
}
if (categories[i] == "c"){
x1_1[i] <- 0
x2_1[i] <- 1
x3_1[i] <- 1
x1_2[i] <- 0
x2_2[i] <- 1
x3_2[i] <- 0
}
if (categories[i] == "d"){
x1_1[i] <- 1
x2_1[i] <- 0
x3_1[i] <- 0
x1_2[i] <- 0
x2_2[i] <- 1
x3_2[i] <- 1
}
if (categories[i] == "e"){
x1_1[i] <- 1
x2_1[i] <- 0
x3_1[i] <- 1
x1_2[i] <- 1
x2_2[i] <- 0
x3_2[i] <- 0
}
if (categories[i] == "f"){
x1_1[i] <- 1
x2_1[i] <- 1
x3_1[i] <- 0
x1_2[i] <- 1
x2_2[i] <- 0
x3_2[i] <- 1
}
if (categories[i] == "g"){
x1_1[i] <- 1
x2_1[i] <- 1
x3_1[i] <- 1
x1_2[i] <- 1
x2_2[i] <- 1
x3_2[i] <- 0
}
if (categories[i] == "h"){
x1_1[i] <- 0
x2_1[i] <- 0
x3_1[i] <- 0
x1_2[i] <- 1
x2_2[i] <- 1
x3_2[i] <- 1
}
}
L1 <- lm(y ~ x1_1 + x2_1 + x3_1) # R^2 = 0.2344, p < 2.2e-16
L2 <- lm(y ~ x1_2 + x2_2 + x3_2) # R^2 = 0.9844, p < 2.2e-16
L3 <- lm(y ~ categories) # R^2 = 0.9844, p < 2.2e-16
By coincidence, the coding that I did as an alternative to the coding in the OP resulted in the same $R^2$ as the standard way of encoding the categorical variable. However, I had no way of knowing that when I started doing the alternative coding.
It is worth a mention that, when I change the y variable, that coincidence vanishes. Additionally, the p-values for the overall $F$-test differ between all three models. (This was probably true with the first simulation, but the p-values were so small that R cannot distinguish between them.)
set.seed(2022)
N <- 100
categories <- rep(c("a", "b", "c", "d", "e", "f", "g", "h"), rep(N, 8))
y <- rnorm(length(categories))
x1_1 <- x2_1 <- x3_1 <- x1_2 <- x2_2 <- x3_2 <- rep(NA, length(y))
for (i in 1:length(y)){
if (categories[i] == "a"){
x1_1[i] <- 0
x2_1[i] <- 0
x3_1[i] <- 1
x1_2[i] <- 0
x2_2[i] <- 0
x3_2[i] <- 0
}
if (categories[i] == "b"){
x1_1[i] <- 0
x2_1[i] <- 1
x3_1[i] <- 0
x1_2[i] <- 0
x2_2[i] <- 0
x3_2[i] <- 1
}
if (categories[i] == "c"){
x1_1[i] <- 0
x2_1[i] <- 1
x3_1[i] <- 1
x1_2[i] <- 0
x2_2[i] <- 1
x3_2[i] <- 0
}
if (categories[i] == "d"){
x1_1[i] <- 1
x2_1[i] <- 0
x3_1[i] <- 0
x1_2[i] <- 0
x2_2[i] <- 1
x3_2[i] <- 1
}
if (categories[i] == "e"){
x1_1[i] <- 1
x2_1[i] <- 0
x3_1[i] <- 1
x1_2[i] <- 1
x2_2[i] <- 0
x3_2[i] <- 0
}
if (categories[i] == "f"){
x1_1[i] <- 1
x2_1[i] <- 1
x3_1[i] <- 0
x1_2[i] <- 1
x2_2[i] <- 0
x3_2[i] <- 1
}
if (categories[i] == "g"){
x1_1[i] <- 1
x2_1[i] <- 1
x3_1[i] <- 1
x1_2[i] <- 1
x2_2[i] <- 1
x3_2[i] <- 0
}
if (categories[i] == "h"){
x1_1[i] <- 0
x2_1[i] <- 0
x3_1[i] <- 0
x1_2[i] <- 1
x2_2[i] <- 1
x3_2[i] <- 1
}
}
L1 <- lm(y ~ x1_1 + x2_1 + x3_1) # R^2 = 0.005842, p = 0.1979
L2 <- lm(y ~ x1_2 + x2_2 + x3_2) # R^2 = 0.001438, p = 0.7659
L3 <- lm(y ~ categories) # R^2 = 0.0123, p = 0.1981
The problem is not to encode the different id values of the categories.
if there are 8 categories, why don't we code them as
0 0 1 0 1 0 0 1 1 1 0 0 1 0 1 1 1 0 1 1 1?
Instead, the problem is to encode the different effects of the categories.
Your suggestion is to use only three dummy variables to model the effect as
$$\text{effect of category $=$ $\alpha_1$ (if category $=$ A, C, E, G) + $\alpha_2$ (if category $=$ A, B, E, F) + $\alpha_3$ (if category $=$ A, B, C, D)}$$
In this way you can indeed encode eight category values by a binary 3 bit binary number.
If we choose $\alpha_1 = 1$, $\alpha_2 = 2$ and $\alpha_3 = 4$ then we can get the values of the 8 id values
$$\begin{array}{} A & 1\alpha_1 + 1\alpha_2 + 1\alpha_3 = 7 \\ B & 0\alpha_1 + 1\alpha_2 + 1\alpha_3 = 6 \\ C & 1\alpha_1 + 0\alpha_2 + 1\alpha_3 = 5 \\ D & 0\alpha_1 + 0\alpha_2 + 1\alpha_3 = 4 \\ E & 1\alpha_1 + 1\alpha_2 + 0\alpha_3 = 3 \\ F & 0\alpha_1 + 1\alpha_2 + 0\alpha_3 = 2 \\ G & 1\alpha_1 + 0\alpha_2 + 0\alpha_3 = 1 \\ H & 0\alpha_1 + 0\alpha_2 + 0\alpha_3 = 0 \\ \end{array}$$
But we are not interested in modelling the id-values 7,6,5,4,3,2,1,0. Instead we want to model the entire possible space of effects. That space is 8-dimensional and not 3-dimensional.
0 0 0and then your codes, you would be suggesting that the difference between $D$ and $A$ was in effect a combination of the difference between $B$ and $A$ and the difference between $C$ and $A$ – Henry Oct 26 '22 at 13:15