Intelligently selecting outliers

Question

I'm trying to remove what might be considered "unreasonable" data by evaluating the percent error in the mean and square root of the variance. Here's the setup:

Let's say I have three bids on a contract. The contractors' total bids are all relatively close. But the itemized breakdown of the bids can have extremely high variances in them.

For example:

  # Total Bid  Item 1  Item 2  Item 3  Item 4  Item 5
  - ---------  ------  ------  ------  ------  ------
  1 827,558    1,026   27.7    800     1,000   1,998
  2 667,118    950     25      80      3,000   23
  3 720,909    1,100   25      25      1,100   22.4
--- ---------  ------  ------  ------  ------  ------  
err 9.03       5.97    4.91    117     54.1    136.78

The "err" is the percentage error between the mean and the square root of the variance of each group, calculated as:

((mean - var^(1/2)) / mean) * 100

This metric does a great job representing the problem that I think I need to address. For example, the % error of Items 1 and 2 show that the bidders bid pretty consistently. It also indicates that the item bids were more consistent than the overall bid totals (error 9.03%).

By contrast, Items 3 - 5 show a higher degree of inconsistency, ranging from 54% to over 136%.

Here's what I know about the data a priori:

The high bids of Item 3 and Item 5 are garbage. By that, I mean, there's no real way to have anticipated those bids. It's just the bidder playing games with how they itemize their bids (really high on one item, really low on another) to mitigate extra costs if they get awarded the contract. In both Items 3 and 5, the lower bids are far closer to the value of the work.

Item 4 has a more ambiguous distribution. It could be that the lower bids represent the value of the work more accurately (and likely, they do), but it may also be the value is higher here than it seems. I might be reluctant to throw out the high bid and maybe consider a weighted average as the real value of the work.

I should also point out, that I'm using this data with a neural network. Ideally, the model's prediction error would be 15% or less.

So, in order to treat this as conservatively as possible, keeping outliers that might reasonably contribute to the model while throwing out ones that are obviously useless, I've considered a couple of approaches:

1. Reject all bids for an item if the item's % error exceeds a set threshold.

2. Reject only the most variant bids when % error exceeds the threshold.

It seems to me the best approach might be #1, using a threshold that scales with the desired error of the model...

Answer 1

After taking some time to investigate the topic of robust statistics more thoroughly, I've opted for a better, although not ideal, way to select my outliers.

This post identifies the Mean Absolute Deviation (MAD) as an excellent robust alternative to mean variance. Of course, it's application is univariate and my use case is multivariate.

Thus, I dug a little deeper and discovered the rrcov package in R. It works nicely, providing distance plots for multivariate data. It identified those points I knew a priori to be outliers, as well as a good number of points that I could have not otherwise identified.

The implementation I'm opting for, then, is to perform this multivariate distance analysis using the tools provided by the rrcov library. Having reduced the data to a series of distances from the median, I can then apply a univariate technique like MAD to select the least variant (or "nearest") data points.

A few caveats that I have identified:

1. The ability to detect outliers is limited by how well the features of the data set describe the dependent variable. Thus, I cannot assume that, just because I've used a multivariate technique, that I'm detecting "true" outliers. Conversely, as I discover other features which may better describe the dependent variable, I can expect my outlier detection to improve as well.
2. While robust statistics like MAD tolerate the use of cut-off rules, this requires certain assumptions about the data that do not apply in my case. Thus, I've opted for simply taking a fixed percentage of the "nearest" data points (e.g. 95%), noting that the remainder left out will be the most variant for the data features I've chosen.
3. MAD assumes a symmetric data distribution. I don't have that, unfortunately. As a result, I'm using the S estimator (see Wikipedia) which employs the relative difference between datapoints and is, therefore, less susceptible to asymmetric distributions.

This question (and user603's fantastic insights) have given me a much better understanding of robust statistical methods, and I am certainly more comfortable with the idea of removing outliers. It may be true that every point is "there for a reason", but until I can adequately describe that reason (as relevant dataset features), it simply is not a useful datapoint.