3

I have lists of scores, which can be very non linear. For example:

20332
18000
599
39
19
2

I need to normalize these scores to a 1-100 scale so I can do some comparisons with other score sets which may be in different scale, such as:

294
39
8
5
3
1

The thing is that, due to the non linearity, I want to give the high outliers less relative weight. So the scores above will normalize to something closer to a rank. For example:

100
60
18
12
7
1

What is the best way to do normalization in such a case?

Ben
  • 124,856
Nir
  • 253
  • A simple method is to just compute the empirical CDF & rescale it. Here's an example: https://stats.stackexchange.com/a/380284/22311 – Sycorax Jun 15 '22 at 18:17
  • 3
    This sounds mighty arbitrary. If you would like it to have some meaning, please explain what these scores might be measuring and what the comparison is supposed to reflect. – whuber Jun 15 '22 at 18:31
  • 1
    I’m with whuber that we will need a lot more detail to be able to address this in a way that gives meaning to your normalized scores. – Dave Jun 15 '22 at 18:39

1 Answers1

4

Firstly, are you certain you want to transform to 1-100, as opposed to 0-100? (The latter scale would be much more sensible in most contexts.) Assuming you actually want to transform to the latter scale, if your data is $\mathbf{x}=(x_1,...,x_n)$ one option would be to define the empirical distribution function:

$$F_\mathbf{x}(x) = \frac{1}{n} \sum_{i=1}^n \mathbb{I}(x \leqslant x_i),$$

and then use a transformation of the form:

$$y_k = 100 \times\frac{n F(x_k) - \alpha}{n+1-2\alpha},$$

where $0 \leqslant \alpha \leqslant 1$ is a shifting parameter used for the transformation (a good value here is $\alpha = \tfrac{1}{2}$). The values $y_k$ are essentially just scaled sample quantiles of the corresponding values in $\mathbf{x}$. If you were to use a transformation of this kind then comparisons of points across different data sets would essentially be comparing the sample quantiles of corresponding points.

As whuber points out in the comments, you might want to first give more thought to why you want to transform your observed data onto this scale, and what kind of subsequent analysis you propose to do. Transformation using the method I'm showing here is essentially conversion to look at the estimated quantiles of the data values (on a 0-100 scale) so it is one meaningful way to do this. You'll have to consider whether or not this is the meaning you want for your scale and what you plan to use it for.

Ben
  • 124,856