6

I am analysing big samples of integer values. Because they come from a non-symmetrical distribution (high positive skew), I prefer to use the median to characterise central tendency, not the mean. However, due to the nature of the data, many values are equal to the median, meaning that median is TOO stable and insensitive to changes in data.

As a toy example, let A = [1 1 1 2 2 2 3] and B = [1 2 2 2 3 3 3].

Both median(A) and median (B) are 2; however intuitively I'd say that median(A) is 'low 2' and median(B) is 'high 2'.

In a more realistic example, we'd have million values of 1, couple of million 2's and the a long dwindling tail to, say, maximal value of 1000.

I wonder, are there any extensions of the definition of median that take into account this difference?

Mark Shovman
  • 163
  • in the case of this toy data, and even more so, in the case of my real data, they are both 2... perhaps my question should be - what to do when there is a long stretch of equal order statistics? – Mark Shovman Dec 22 '14 at 12:56
  • 2
    One could invent all sorts of extensions, but more guidance would be useful. For instance, an important property of the median for many purposes--including for description and exploration--is the tendency of the median to be equivariant under monotonic transformations. For instance, the logarithm of the median (of an odd-size dataset of positive values) always equals the median of the logs of the data. If equivariance is a desirable property, your options will be more limited. Generally, any extension of the median in the way you describe may require some distributional assumptions. – whuber Dec 22 '14 at 17:50
  • 1
    Equivariance is definitely a desirable property, thanks for clarifying.The distribution is positive, long-tail, reasonably power-law - mind, this is real-life data, so dirty. – Mark Shovman Dec 23 '14 at 09:25

2 Answers2

6

The trimmed mean is, from where you are starting, one generalisation of the median. If you trim (meaning, ignore rather than drop) 3 values in each tail of an ordered sample of 7 then you get the median; if you trim 0 values, then you get the mean. For small samples, thinking in terms of number trimmed is natural. Here is a Stata-based calculation with your "data" using code published with Cox (2013), but the output should be fairly transparent to users of other software: 

set obs 7
mat A = (1, 1, 1, 2, 2, 2, 3) 
mat B = (1, 2, 2, 2, 3, 3, 3)
gen A = A[1, _n]
gen B = B[1, _n]

trimmean A, number(0/3)

+---------------------------+
| number   #   trimmed mean |
|---------------------------|
|      0   7       1.714286 |
|      1   5            1.6 |
|      2   3       1.666667 |
|      3   1              2 |
+---------------------------+

trimmean B, number(0/3)

+---------------------------+
| number   #   trimmed mean |
|---------------------------|
|      0   7       2.285714 |
|      1   5            2.4 |
|      2   3       2.333333 |
|      3   1              2 |
+---------------------------+

As common, results are shown to more decimal places than will be needed.

For larger samples, it is more natural, and certainly conventional, to think in terms of the fraction or percent trimmed. The 25% trimmed mean has been given various names, the most common being "midmean". (Those familiar with box plots can think of it as the mean of the values falling inside the box.)

The advantages of trimmed means include

  1. Ease of understanding and calculation. Trimmed means are used in judging sports as a way of discounting or discouraging bias in voting, so they may even be familiar to users of statistics from outside the field.

  2. Clear links to standard ideas, mean and median.

  3. Flexibility in choosing that mix of resistance to wild values and use of the information in the other values that is a good trade-off in a project.

The disadvantages include

  1. Flexibility is another name for arbitrariness. It's not easy to see what the best extensions to bivariate or multivariate cases would be.

  2. Values are included or not, at least in the simplest flavour of trimmed means, which may not be subtle enough.

  3. Trimmed means other than the limiting cases of mean and median lose many of the attractive properties of either, including the equivariance of median and monotonic transformations emphasised by @whuber.

Cox (2013) is a tutorial review emphasising the history of ideas and associated graphics. (It overlooks a brief mention by Jules Verne.)

Cox, N. J. 2013. Speaking Stata: Trimming to taste. Stata Journal 13: 640-666. http://www.stata-journal.com/article.html?article=st0313

Nick Cox
  • 56,404
  • 8
  • 127
  • 185
  • Thanks! How does trimmed mean work with highly-skewed data? – Mark Shovman Dec 23 '14 at 12:12
  • 1
    As said, you ignore values in the tails, so the trimmed mean is the mean of values in the middle of the distribution. Note that nothing in the definition obliges you to trim equal numbers in either tail, although that's the usual default. – Nick Cox Dec 23 '14 at 12:20
  • 1
    +1: nice idea. Note, however, the lack of equivariance of the trimmed mean. (There might be no way to get around this problem: some property of the median has to be given up in order to generalize it!) – whuber Dec 23 '14 at 14:10
2

I disagree with your characterization of B median as "upper 2", because its mean is 16/7=2.29. You alluded to the fact that you didn't like mean for the distribution is skewed, so characterizing the median as "upper 2" would be inconsistent with the sample mean. Mean of sample A is 1.71. Hence, the the central tendency is probably high 1 and low 2 for samples A and B.

I propose to use a weighted average of mean and median:

$m=w*mean+(1-w)median$.

In your case median = 2, and A and B means are 12/7 and 16/7. So, if you use $w=1/3$, then m=1.9 and 2.1 would be consistent with proposed above high 1 and low 2 characterization. You can play with weights w to get a better metric for your study. High $w$ will make it look more like mean, and low $w$ will make it more like a median.

Aksakal
  • 61,310