0

I am working with a substantial dataset in which I need to compare the distributions of certain common features across different categories. The challenge I face is that due to the imprecision in classification criteria, it often leads to the rejection of the null hypothesis in standard hypothesis tests, even though I believe that with more precise categorization, these distributions should be comparable.

I am aware of the Kolmogorov-Smirnov (KS) test and its statistic 'D', which provides a normalized measure of the distance between two distributions. I'm looking for methods similar to the KS test that offer a normalized parameter. This parameter would enable me to effectively categorize the differences between distributions as "large differences" or "small differences" in a straightforward manner.

Methods like Earth Mover's Distance are not normalized. This lack of normalization prevents me from having a unified criterion for comparing different features.

In essence, my question is: I know these distributions are "significantly" different, but I want to quantify how different they are. Any guidance or suggestions would be greatly appreciated. Thank you!

Song Nyanko
  • 1
  • Are you by any chance looking for an effect size ? It would help to know what test you performed to say that "these distributions are statistically different". – CaroZ Oct 23 '23 at 08:08
  • Thank you @CaroZ, effect size is really helpful. I've used one-way ANOVA, t test, Welch's t test, KS test, Kruskal-Wallis test, and Mann-Whitney U test. – Song Nyanko Oct 23 '23 at 08:41
  • All these different tests on one dataset ? Do you mean that you run a different test on each feature ? – CaroZ Oct 23 '23 at 09:35
  • Yes, I conducted each of these tests on every feature to gain insights into whether their distributions differ significantly across different categories. – Song Nyanko Oct 23 '23 at 10:07
  • When you mention "D", do you mean Cohen's d ? – CaroZ Oct 23 '23 at 10:10
  • No, when I mentioned 'D,' I was referring to the statistic 'D' from the KS test. And I learnt it from here. – Song Nyanko Oct 23 '23 at 10:22

0 Answers0