4

I am working on building a regression model to predict housing sales price using house features (Ames housing dataset). And I prepared feature set in two ways:

Case 1.

I performed Box-Cox transformation on all the numerical features and performed one-hot encoding on the categorical features.

Case 2.

I used the numerical feature set as it is and performed one-hot encoding on the categorical features.

I applied linear regression on both feature sets and got $R^2$ much, much lower in Case 1 (where numerical features were transformed). In Case 2 it was always more than 0.85 for any train test combination and in Case 1 it was never more than 0.5.

Since linear regression assumes the features to be conditionally independent and normal distributed and Box-Cox helps to make features near normally distributed I thought performing Box-Cox will boost my $R^2$. Can anyone help me understand why that plummeted for the model applied to Box-transformed data?

Nick Cox
  • 56,404
  • 8
  • 127
  • 185
Sujit Desai
  • 41
  • 2
    Your understanding of linear regression, unfortunately, is quite wrong. There is no assumption that features should be independent or normal. Maybe the untransformed variant contains strong outliers that can trigger leverage effects, which can affect the R-squared. Working with a clean train/valid/test strategy could reveal what is going on. – Michael M Jun 05 '22 at 21:03
  • 1
    If you transformed the response variable, then the $R^2$ values just aren't comparable. Not even remotely. That's because they compare reductions in the variances of two different sets of data. – whuber Jun 06 '22 at 00:33
  • @MichaelM Thank you for the insight. I did further research and discovered that the normality of the features is not an assumption. In fact, I should have figured that while performing one hot encoding (that it will obviously not result in a perfect or near normally distributed dummy variables all the time). However does not independence of the features is one of the assumptions in a ideal case linear regression? That is what I make of the lack of multi-collinearity assumption from this article: https://en.wikipedia.org/wiki/Linear_regression#Assumptions – Sujit Desai Jun 07 '22 at 05:47
  • @whuber Appreciate your input! By response variable do you mean the dependent variable or output variable (or the Y in Y = B*X). In my case the dependent variable was in consistent format in both the cases. – Sujit Desai Jun 07 '22 at 05:52
  • By "consistent format" would you mean "expressed the same way"--that is, not transformed? If so, the reason for a reduction in $R^2$ is simple: the model using transformed variables is worse than the model without them. This comes down to how linear the relationship is between the explanatory variables and the response. – whuber Jun 07 '22 at 13:16
  • 1
    @whuber, yes in both cases the response variable used in model was log of actual response variable. But I get your point. – Sujit Desai Jun 08 '22 at 04:27
  • For some guidance about transforming regression variables, see https://stats.stackexchange.com/a/3530/919; and for an example of a practical procedure, see https://stats.stackexchange.com/a/35717/919. A maximum likelihood version of the latter is available in the MASS package for R as the function boxcox. – whuber Jun 08 '22 at 12:49

1 Answers1

1

As was mentioned in the comments, there is no normality assumption about the features. It is common to assume a normal error term in linear regression, as this means that the OLS solution for the coefficient estimates is equivalent to maximum likelihood estimation of the regression coefficients and that the usual standard errors are correct, but even that does not require normality of the features.

It is a common misconception that the features should be normally distributed. I think this comes from the normal-errors assumption being abstract, so people make the mistake of assuming normal features.

Consequently, when you transform the features, there is no reason to expect that being normal gets you closer to some ideal that should result in higher performance. You aren’t assured of worse performance, either, but if you transform features that are roughly linearly related to $y$, you might wind up with a relationship between the transformed feature and $y$ that is not linear, and your linear model will miss such a relationship.

EDIT

A simulation shows this in action.

library(MASS)
set.seed(2024)
N <- 1000
x1 <- rchisq(N, 1) # Non-normal feature
x2 <- rchisq(N, 1) # Non-normal feature
e <- rnorm(N)
Ey <- x1 - x2
y <- Ey + e 
t1 <- (x1)^(1/3) # Cube root is the Box-Cox transformation for chi-squared
t2 <- (x2)^(1/3) # https://stats.stackexchange.com/a/565868/247274
L1 <- lm(y ~ x1 + x2) # Fit to original features
L2 <- lm(y ~ t1 + t2) # Fit to transformed features
summary(L1)$r.squared # 0.7854019
summary(L2)$r.squared # 0.5932263
par(mfrow = c(2, 2))
plot(x1, y)
plot(x2, y)
plot(t1, y)
plot(t2, y)
par(mfrow = c(1, 1))

In the plots below, notice that transforming the features leads to a curved relationship between the features and the outcome that will not be captured by a linear regression unless such curvature is explicitly included in the model, such as with splines (or cubing the t features back to the original features that have a linear relationship with y).

Plot the tansformed and untransformed features

Dave
  • 62,186