1

I have a number array and i want to pick only significantly big numbers from that array using some statistical method. I tried calculating the average of the array and then subtracting each numbers from that avg and picked only values which are less than 0. like this. (for this number set correct answer should be 13 and 14 as you can notice).

number array  : 10 10 7 13 10 10 14 10 10  
avg = (10+10+7+13+10+10+14+10+10) / 9  
avg = 10.44  


10.44 - 10 > 0 (greater than zero. don't pick it)  
...  
10.44 - 13 < 0 (less than zero. Pick it).

But it doesn't always provide the correct answer

For example for this number array :- 

 10 10 7 14 10 10 7

the above method doesn't work as expected. It should only choose 14 but since the avg is 9.7 it chooses 10 as well.

So is there any way i can statistically pick the significantly high numbers compared to others from a given array reliably.

For example: 

 10 10 7 13 10 10 14 10 10 --> **13** and **14**

 10 10 7 14 10 10 7 --> **14**
HarshaXsoad
  • 113
  • 3

1 Answers1

1

What you are asking for is a way of detecting outliers. The most well-known rules define outliers based on the number of standard deviations from the mean, with popular threshholds being 2, 2.5, and 3. There is no particular science for deciding how to set the threshold for most real-world problems: the higher the threshold, the fewer outliers you will detect.

You also ask about identifying whether observations are "significantly high". This is a more complex issue. To decide that an outlier is "significantly high" you need to have a model that describes the distribution of values that you would expect to get. This will inevitably be different for different data sets (i.e., you need to work this out for yourself, there can be no magic rule applicable to all problems). The most common approach is to assume the data is generated as draws from a normal distribution, and if one does this you can come up with some more "scientific" rules (e.g., that values more than 1.96 standard deviations above or below the mean are "significant"). But, as your example data contains only positive integers, my guess is that a normal distribution is not going to be appropriate, and you will need to work out an appropriate count distribution, iff you are keen to meaningfully draw conclusions about whether values are "significantly high".

Tim
  • 3,401
  • Ok i'll try what you suggested. I already tried the method of finding outliers using Q3 + (1.5 * (Q3 - Q1)) for number array but didn't gave me a good answer. – HarshaXsoad May 24 '16 at 10:26
  • Thanks. Sorry for the delay. It took me sometime to try it out and confirm it. the value 0.5 worked best for me :). Thanks for the insight. – HarshaXsoad May 29 '16 at 07:29
  • May be now it's bit late to ask but i tried the method in above comment with this data set : - 10 10 14 10 10 7 13 10 11 13 10 10. It is supposed to output the values 13 and 14 as outlires but it only outputs 14 since the upper outer fence is 13.75. How i can fix this. – HarshaXsoad Jun 12 '16 at 17:28