Why is it Bad to Discretize a Continuous Variable?

Question

I am an MBA Student taking courses in statistics. I was doing some reading online and came across Carstensen (2020), which I unfortunately do not have access to.

This article suggests that "categorizing quantitative variable" is bad - but I am not sure why this is. Our professor in our statistics courses showed us examples where a decision tree was fit to some data where the dependent variable was originally continuous, but then "binned" into groups. In his example, this strategy seemed to work fine - but some other references are suggesting that this is bad practice.

Can someone please explain why doing this might not be ideal? I understand that if you make arbitrary bins that this could lead to problems and lose information in the process - but if the bins are made carefully and rigorously tested, why is it bad to "categorize quantitative variables"?

Answer 1

There are two main issues here that militate against discretisation into bins for statistical analysis. The first is that this generally involves some loss of information, since different values are put into the same bin, and so there is a corresponding loss of statistical power in the analysis. The second is that such an approach fails to produce inferences that change in a continuous manner with the continuous variable of interest; instead the inferences change over the bins and so they appear as "jumps" when looked at on the original continuous scale. Neither of these problems are necessarily fatal, particularly if you have a reasonably large number of bins to reduce the loss of information.

Ideally, we would be able to model continuous data with models that treat them as continuous. However, continuous models generally involve some specification of a parametric family of distributions that limits the generality of analysis, and this means that there are distributional assumptions that might or might not fit the data well. (By contrast, in the discrete case the multinomial distribution fits all possible probability distributions for the number of outcomes occurring over a finite, or even countably infinite, set of finite bins.) It is usually possible to fit a continuous model that works well on the data if you have an expansive set of tools and the ability to generalise continuous models when they don't fit well. The difficulty that some analysts encounter is that they may find that standard continuous models they use for inference (e.g., linear regression) aren't fitting their continuous variables well and they may reach the limit of their ability to generalise these models effectively to improve things. In such cases, analysts sometimes fall back on the discretisation into bins in order to allow them to apply highly general discrete models (e.g., the multinomial) that do not assume any particular distributional forms. This is a trade-off --- you lose some information and the ability to make smooth inferences about a continuous variable, but you have a high level of generality of the remaining discretised variable and it falls within well-known model forms that are easy to apply.

I note your proposal that discretisation might be okay "...if the bins are made carefully and rigorously tested...". It is easy to say that, but how do you propose to test them (against what), if not by comparing them to an initial analysis of the continuous variable? If you "rigorously test" the discretised method and inference by comparing it to a continuous model that is taken to be the correct analysis, then you already presumably have a well-fitted continuous model, so what is the point of the discretisation? If you can't get that comparator, then what exactly is your proposed "rigorous test" doing? It is not necessarily impossible to answer these questions sensibly, but you would need to think more about what exactly you propose to do and how it helps you.

The above gives you a practical idea of the issue, but I would be remiss if I did not expand on this with a little excursion into the philosophy and foundations of mathematics and computing. In particular, it is worth noting that all continuous data is discretised to some extent in statistical analysis, due to the finite precision of computational representation of numbers. At best we represent continuous variables up to some finite level of precision (e.g., standard precision under floating-point arithmetic) and so we always implicitly use a discretised scale where the "bins" are tiny intervals that are too small to be differentiated under the computational representation. (Usually these bins are precise enough to avoid duplicate values of the "continuous" variable.) Since this is a necessity of analysis, there cannot be any in principle objection to some discretisation of continuous variables in analysis, and so the question becomes one of degree. To take a deeper excursion into the philosophy of mathematics, finitists like Doron Zeilberger would go further with this representation argument and object even to the assertion that continuous random variables, continuous functions, or infinite sets exist; they would say that all purportedly continuous variables are actually discrete, up to the finite level of accuracy of the computational representation, and so the real question is only whether we want to aggregate smaller bins into larger bins.

Answer 2

Others answers have discussed how discretization throws away information, which can hide effects which would be discovered if the continuous data were used. But sometimes the loss of information actually creates illusory effects in the data!

For example, suppose I'm trying to determine if a particular Northern species of bird migrates south in the winter. In the first week of November, I tag each bird caught and released from a research site at, say, $47.5^\circ$ North latitude. To get a larger sample size, I also tag the birds from a more northern site, at $51^\circ$ North latitude. Then I use the tags to locate the birds in December, seeing if they trend south as it gets colder.

Suppose that in reality, the birds don't migrate at all, but just mill about, some heading south, some north. The continuous data would reveal that there is no migratory trend whatsoever, only random displacements between November and December. But I decide (foolishly) to discretize my data into bins $5^\circ$ across. None of the birds I tagged made it below $40^\circ$ or above $60^\circ$, so I break my sample range into four bins, $40^\circ$-$45^\circ$, $45^\circ$-$50^\circ$, $50^\circ$-$55^\circ$, and $55^\circ$-$60^\circ$. This feels like a natural discretization to me, as the ranges have nice, round-number endpoints.

The first site is in the center of a range, so it doesn't cause me any problems. Most of the birds tagged at the $47.5^\circ$ site stay in the $45^\circ$-$50^\circ$ bin, but some of the outliers end up in both the $40^\circ$-$45^\circ$ and $50^\circ$-$55^\circ$ bins. However, the $51^\circ$ site is close to the bottom of its bin, so birds tagged there are much more likely to randomly move to the bin below than the bin above. I notice this pattern in my data, and wrongly conclude that the northern population of birds tend to travel south during November.

When I publish my results in an esteemed ornithology journal, I don't include the latitudes at which the birds were tagged, only the bin in which they were tagged. I do include statistical tests, including a $p$-value below 0.05, which supports the claim that the birds trend into southern bins during Novemeber. My readers have no way of knowing that the reason for this trend is that my discretization biased my methodology, and are convinced that I have discovered a real phenomenon.

Answer 3

A different issue is that the categorical model may be unnecessarily hard to fit, caused by the philosophical mismatch of the model to the problem.

Suppose I have a large spring. The farther I pull it, $x$, the more force it exerts, $y$. This is an extremely straightforward continuous relationship -- pull a bit farther, feel a bit more force. I could fit a model $y = ax$ with just one parameter, $a$. If Hooke's law applies, then just a couple data points will give me a very accurate estimate of $a$.

Now consider the categorical approach. I could break distances up into one hundred buckets, $x \in [0,1]$, $x \in [1,2]$, ..., $x \in [99,100]$. Now I have a model with at least one hundred parameters. It is capable of modeling many relationships that I know are impossible, such as the force going up and back down and back up etc. as the spring is stretched farther and farther. I may need hundreds or thousands of data points to fit the model. And it still won't predict very accurately at resolutions below $1$.

To fix this, I could just use a couple buckets, $x \in [0,50]$ and $x \in [50,100]$. Now I don't need as much data and my model is simple, but very inaccurate.