When to normalize data in regression?

Question

Under what circumstances should the data be normalized/standardized when building a regression model. When i asked this question to a stats major, he gave me an ambiguous answer "depends on the data".

But what does that really mean? It should either be an universal rule or a check list of sorts where if certain conditions are met then the data either should/ shouldn't be normalized.

It should either be an universal rule or a check list of sorts where if certain conditions are met then the data either should/ shouldn't be normalized. Can you justify that? — Matthew Drury, Mar 15 '16 at 21:45
Are you asking about standard linear regression, or about penalized methods like ridge regression or LASSO? — EdM, Mar 15 '16 at 21:46
@MatthewDrury: What i mean is either data should be normalized for building all regression models (OLS, Logistic etc) or it should be done when so and so conditions are not satisfied like non-constant variance..etc (hypothetically speaking) — Raj, Mar 15 '16 at 21:50
This page has an extensive discussion of situations in which normalization/standardization might be important. Please look it over and edit your question to include any issues that you believe weren't discussed adequately there. — EdM, Mar 15 '16 at 21:59
Much software will automatically standardize the variables for its internal calculations. Could you clarify, then, whether you are asking about how to compute numerical solutions or about whether it is actually necessary to standardize the variables before applying regression software? — whuber, Mar 15 '16 at 22:47
See also When and how to use standardized explanatory variables in linear regression, Variables are often adjusted (e.g. standardised) before making a model - when is this a good idea, and when is it a bad one?, & Is centering a valid solution for multicollinearity? — Scortchi - Reinstate Monica, Mar 16 '16 at 14:16

kjetil b halvorsen · Accepted Answer · 2018-07-08T17:34:19.593

Sometimes standardization helps for numerical issues (not so much these days with modern numerical linear algebra routines) or for interpretation, as mentioned in the other answer. Here is one "rule" that I will use for answering the answer myself: Is the regression method you are using invariant, in that the substantive answer does not change with standardization? Ordinary least squares is invariant, while methods such as lasso or ridge regression are not. So, for invariant methods there is no real need for standardization, while for non-invariant methods you should probably standardize. (Or at least think it through).

The following is somewhat related: Dropping one of the columns when using one-hot encoding

by OLS being "invariant", you mean that it is "scale invariant", where scale is the range of the numbers in the predictors/columns fed to OLS? — Naveen Reddy Marthala, Mar 30 '23 at 10:39

mdewey · Answer 2 · 2016-03-16T13:54:47.567

7

It sometimes makes interpretation easier if you subtract the mean or some number within the range of the actual values as this can make the intercept more meaningful. For instance if you have people aged 65 and over subtract 65 and then the intercept is the predicted value for a 65-year old rather than a neonate. If you have non-linear terms like powers this makes them less correlated and so you can see more easily what is going on. It also may make life easier to scale the predictor so as to move the coefficients into a more printable range. For instance converting days into weeks or months. Other than that it should not matter. I suppose some of what I have just written may be what your friend meant by it depends on the data.

edited Mar 16 '16 at 13:54

answered Mar 15 '16 at 21:59

mdewey

17,806

4

The correlation between predictors is unaffected by subtracting means, so nothing is less correlated than before. – Nick Cox Mar 16 '16 at 08:17
You are right @Nick, apologies. It is only non-linear terms for which it helps. – mdewey Mar 16 '16 at 11:32
"non-linear terms like powers this makes them less correlated" this is a general truth because to me it does sounds fine the connection between correlation and the non linearity. Would not a variable in a power be fully correlated with a response Y? I ask this because after this infer that the standardization in a ridged regression linear model sounds unnecessary. Thank you – nikolaosmparoutis May 10 '19 at 14:58
@NickCox: How about when dividing in case we want to standardize. Don't we get unitless coefficients? – MSIS Dec 19 '19 at 00:17

When to normalize data in regression?

2 Answers2

Linked