Overall rank from multiple ranked lists

Question

I've looked through a lot of literature available online, including this forum without any luck and hoping someone can help a statistical issue I currently face:

I have 5 lists of of ranked data, each containing 10 items ranked from position 1 (best) to position 10 (worst). For sake of context, the 10 items in each lists are the same, but in different ranked orders as the technique used to decide their rank is different.

Example data:

            List 1      List 2      List 3     ... etc
Item 1     Ranked 1    Ranked 2    Ranked 1     
Item 2     Ranked 3    Ranked 1    Ranked 2
Item 3     Ranked 2    Ranked 3    Ranked 3
... etc

I am looking for a way to interpret and analyse the above data so that I get a final result showing the overall rank of each item based on each test and its position, e.g.

Result
Rank 1 = Item 1
Rank 2 = Item 3
Rank 3 = Item 4
... etc

So far I have attempted to interpret this information from performing Pearson's Correlation, Spearman's Correlation, Kendall Tau's B, and Friedman tests. I have found however, that these results have generally paired my lists (i.e. compared list 1 to list 2, then list 1 to list 3 .. etc), or have produced results such as Chi-Square, P-Values etc about the overall data.

Does anyone know how I can interpret this data in a statistically sound method (at a post graduate / PhD applicable level) so that I can understand the overall ranks signalling the importance of each item in the list across the 5 tests please? Or, if there is another type of technique or statistical test I can look into I would appreciate any hints or guidance.

(It maybe also worth noting, I have also performed the simpler mathematical techniques such as sums, averaging, minimum - maximum tests etc, but do not feel these are statistically important enough at this level).

Any help or advice would be greatly appreciated, thank you for your time.

Answer 1

I am not sure why you were looking at correlations and similar measures. There doesn't seem to be anything to correlate.

Instead, there are a number of options, none really better than the other, but depending on what you want:

Take the average rank and then rank the averages (but this treats the data as interval)

Take the median rank and then rank the medians (but this may result in ties)

Take the number of 1st place votes each item got, and rank them based on this

Take the number of last place votes and rank them (inversely, obviously) based on that.

Create some weighted combination of ranks, depending on what you think reasonable.

Answer 2

As others have pointed out, there are a lot of options you might pursue. The method I recommend is based on average ranks, i.e., the first proposal of Peter.

In this case, the statistical importance of the final ranking can be examined by a two-step statistical test. This is a non-parametric procedure consisting of the Friedman test with a corresponding post-hoc test, the Nemenyi test. Both of them are based on average ranks. The purpose of the Friedman test is to reject the null hypothesis and conclude that there are some differences between the items. If it is so, we proceed with the Nemenyi test to find out which items actually differ. (We don't directly start with the post-hoc test in order to avoid significance found by chance.)

More details, such as the critical values for these both tests, can be found in the paper by Demsar.

Answer 3

I (well, Google) found a paper that benchmarks methods for combining ranked lists:

Li, X., Wang, X. and Xiao, G., 2019. A comparative study of rank aggregation methods for partial and top ranked lists in genomic applications. Briefings in bioinformatics, 20(1), pp.178-189. https://doi.org/10.1093/bib/bbx101

They use two R packages: TopKLists: https://cran.r-project.org/web/packages/TopKLists/index.html RobustRankAggreg: https://cran.r-project.org/web/packages/RobustRankAggreg/index.html

Answer 4

Use Tau-x (where the "x" refers to "eXtended" Tau-b). Tau-x is the correlation equivalent of the Kemeny-Snell distance metric -- proven to be the unique distance metric between lists of ranked items that satisfies all the requirements of a distance metric. See chapter 2 of "Mathematical Models in the Social Sciences" by Kemeny and Snell, also "A New Rank Correlation Coefficient with Application to the Consensus Ranking Problem, Edward Emond, David Mason, Journal of Multi-Criteria Decision Analysis, 11:17-28 (2002).

Answer 5

It seems to me that the unstated assumptions in the question as originally posed were that the sources of all five lists are to be regarded as equally credible and are to be given equal weight. In that case the problem is equivalent to that of determing the winner(s) in an election where the individual votes consist of ranking the candidates in order of preference.

A sound aggregate ranking may often be constructed using a pairwise comparison of items. For example, if three out of five lists rank item A above B, while the other two rank B above A, then the aggregate should rank A above B; similarly for each pair of items. Unfortunate it is not always possible to construct such an aggregate list without circular relations (eg. the majority rank A above B, B above C, but C above A); in voting analysis this is known as the Condorcet paradox. See the Wikipedia article: https://en.wikipedia.org/wiki/Condorcet_paradox (retrieved 2023-11-19)

In the example provided in the question, just taking the three lists and the three items shown, a consistent aggregate ranking would be:

• Rank 1... Item 1
• Rank 2... Item 2
• Rank 3... Item 3

This is consistent because more individual lists rank item 1 over item 2 than the reverse, therefore 1 is above 2 in the aggregate ranking... and so on for each pairwise comparison. In general however it cannot be guaranteed that such a consistent aggregate ranking exists; the aforementioned circular relation can occur with as few as three items. In the event of such a circular relation it is necessary to have some extra means at hand of deciding which of the pairwise comparisons are to faithfully followed and which are to be ignored, and this cannot be done without more information about the sources of the lists and the means by which the individual rankings were constructed.

In summary: depending on the individual lists, sometimes it is possible to construct a reliable aggregate ranking, while in other cases it is not possible without further knowledge of the source of the individual rankings, or assumptions about those sources.