0

I have a dataset with all the calls to an emergency service in a period of time.

I have information about the time (in seconds) that it took to respond to that call and the area where the emergency took place.

The emergency service admitted that some records have been recorded incorrectly. For example, there are response times that last for 24 hours or just 0 seconds. This mistakes are caused by human errors but they can compromise my analysis.

I want to find out the average response time in each area but, as there are a lot of outliers, I want to use the trimmed mean for each area.

My question is: is there any other way of excluding the outliers for the whole dataset instead of doing it for each area?

Andy
  • 19,098
Duarte_RV
  • 103

1 Answers1

0

If you know what values (not what % of the values, but what actual values) are outliers, you can delete them from the whole data set. However, if you want a trimmed mean for each group, you have to take the trimmed mean for each group. Which is better probably depends on the nature of the errors; you'd want to look at the distributions to see if some times are simply way above or below any reasonable value. E.g if there are a lot of 24 hours, delete them. Similarly, there might be some that are explicable errors (depending on how the times were recorded) - e.g hours instead of minutes.

The problem is that human error can probably cause inliers as well as outliers, but I don't see a way around that.

Peter Flom
  • 119,535
  • 36
  • 175
  • 383
  • I would be concerned about using the "just delete them" procedure until I had investigated whether the obviously erroneous values might correspond to special circumstances. If, for instance, they all corresponded to extremely fast response times, then deleting them would create a positive bias in the estimates. At a minimum, the numbers and proportions of data that are deleted (or otherwise specially handled in some way) should be documented and the potential effects on the analysis should be evaluated. – whuber Feb 16 '15 at 15:26
  • I can't "just delete them" because the dataset contains about 20 million records, so it's not possible to do it manually. I will take the trimmed mean for each group and I will write a disclaimer explaining why I did it and the limitations of this choice. This is not for a statistical publication, it is a journalistic piece so I just want it to be accurate. Thank you both for your help! – Duarte_RV Feb 16 '15 at 16:45
  • 1
    @whuber raises a good point (as usual). Also, you can just delete them automatically. You didn't say what package you are using, but this would be easy in R or SAS or (I am pretty sure) any other package. – Peter Flom Feb 16 '15 at 17:26