Does the presence of an outlier increase the probability that another outlier will also be present on the same observation?

Question

**Edit: (10/26/13) More clear (hopefully) mini-rewrites added at the bottom**

I'm asking this from a theoretical/general standpoint - not one that applies to a specific use case.

I was thinking about this today:

Assuming the data does not contain any measurement errors, if you're looking at a specific observation in your data and one of the measurements you recorded contains what could be considered an outlier, does this increase the probability (above that of the rest of the observations that do not contained measured outliers) that the same observation will contain another outlier in another measurement?

For my answer I'm looking for some sort of theorem, principle, etc. that states what I'm trying to communicate here much more elegantly. For clearer explanations see Gino and Behacad's answers.

Example:

Let's say you're measuring the height and circumference of a certain type of plant. Each observation corresponds to 1 plant (you're only doing this once).

For height, you measure:

Obs 1  |   10 cm
Obs 2  |   9 cm
Obs 3  |   11 cm
Obs 4  |   22 cm
Obs 5  |   10 cm
Obs 6  |   9 cm
Obs 7  |   11 cm
Obs 8  |   10 cm
Obs 9  |   11 cm
Obs 10  |   9 cm
Obs 11  |   11 cm
Obs 12  |   10 cm
Obs 13  |   9 cm
Obs 14  |   10 cm

Since observation 4 contains what could be considered an outlier from the rest of the data, would the probability increase that measured circumference also contains an outlier for observation #4?

I understand my example may be too idealistic but I think it gets the point across...just change the measurements to anything.

Edited in attempts to make more clear:

(10/26/13)

Version 1 Attempt of Abbreviated Question:

In nature and in general, is there a tendency (even a weak one) that the probability is greater that the "degree of variance from the mean in any attribute(s) of an observation" will be similar to the "degree of variance from the mean in any other* specific attribute of that same observation" in comparison to the probability that it will INSTEAD be more similar to the "degree of variance from the mean in that same* specific attribute of any other observation."

* next to a word means I was pairing what they reference.
"Quotes" used above mean nothing and are used simply to help section parts together/off for clarity.

Version 2 Attempt of Abbreviated Question:

In nature and in general, is variance from the mean across observations for one attribute¹ correlated (even with extremely loose correlation) to the variance from the mean across observations for all attributes¹?

¹Attribute meaning measurement, quality, presence-of-either of these, and/or nearly anything else that the word "attribute" could even slightly represent as a word. Include all synonyms of the word "attribute" as well.

Answer 1

It might be true for your example (or it might not—plant No. 4 could be an etiolated seedling elongating its stem in a hunt for light without expanding its girth), but it's a matter of prior botanical knowledge rather than any general statistical law. It's not hard to construct counter-examples: someone who's an outlier on hours per week spent exercising is less likely to be an outlier on blood pressure. Any type of dependence is possible in principle—Chaos Theory doesn't say the butterfly that flaps its wings hardest causes the biggest hurricane (I don't know what Zen Buddhism says). A general theory that ranges over all measurements anyone might want to make on all kinds of things is going to be elusive.

If you consider a world with a well-defined set of objects, $a_1, a_2, \ldots$, & a well-defined set of measurements we can make on each object, $x_{11}, x_{12}, \ldots, x_{21}, x_{22}\ldots$, then your question could perhaps be framed in an answerable way ("Pick $n$ objects at random from the $a$s, pick two measurements at random from the $x$s, ..."); for our world I don't see how it can be.

Answer 2

NARROW VIEW 1
A narrow view of this question (since the other answers and comments I believe have covered adequately various general approaches) is "Assume two random variables $X$ and $Y$ are dependent. Does the variance of $X$ conditional on $Y$ is a function of the variance of $Y$?

To take refuge in the normal distribution, if both $X$ and $Y$ are normal and dependent, denote $\sigma^2_x$, $\sigma^2_y$, $\sigma_{xy}$ their unconditional variances and their covariance respectively. Then

$$\operatorname {Var}(X\mid Y) = \sigma^2_x-\frac {\sigma_{xy}^2}{\sigma^2_y} $$

which is increasing in $\sigma^2_y$. Note that the direction of their covariance (positive/negative) doesn't matter.

But also, note that the conditional variance is lower than the unconditional variance...

NARROW VIEW 2
A 2nd narrow view of the matter is: consider pairs of random variables $(X_i,Y_i),\; i=1,...,n$, and $\sigma^2_x(i)$ and $\sigma^2_y(i)$ their unconditional variances. Assume that for some indices $k,j \in [1,...,n]$, we have $\sigma^2_y(k) > \sigma^2_y(j), \forall j\neq k$.
Should we "expect" that $\sigma^2_x(k) > \sigma^2_x(j), \forall j\neq k$ also?
This formalization of the OP's question requires from us to consider directly the dependence of each $(X_i,Y_i)$ on a common source, say a vector of random variables $\mathbf Z_i$, So we have

$$X_i = h_i(\mathbf Z_i),\;\; Y_i = g_i(\mathbf Z_i)$$

Then we ask "does (or when) $$ \operatorname {Var}[g_k(\mathbf Z_k)]>\operatorname {Var}[g_j(\mathbf Z_j)] \Rightarrow ? \operatorname {Var}[h_k(\mathbf Z_k)]>\operatorname {Var}[h_j(\mathbf Z_j)],\; \forall j\neq k$$

To be able to tell something about such an inequality, it must be the case that at least some of the variables that appear in $Z_j$, must also appear in the other $Z_i$'s: then as a necessary condition, we do not only require that the elements of each pair of $(X_i,Y_i)$ rv's are dependent -we require also that the pairs are dependent between them: good-bye i.i.d samples...

Answer 3

I've thought about this question for a year and had an epiphany a few days ago.

In another question I left an answer to, @FrankHarrell left a separate answer that I to this day still find quite profound.

Here: https://stats.stackexchange.com/a/71748/28928

He says "Variable selection without penalization is invalid."

If you're attempting to analyze any set of data and determine the influence of variables on each other, then, really, the only variables you're using are the ones you have in your dataset - even if you discover that there is no relationship between certain variables.

Let's say X(sub 5) has what seems to be absolutely no relationship to your Y - so you discard it.

If you discard it, though, then you're selecting variables...incurring penalization. Additionally, if you're missing ANY variables in your dataset, you're selecting variables once again. When I say ANY, I literally mean any and every variable in the universe.

And, well, "Variable selection without penalization is invalid."

Thus, if

"Variable selection without penalization is invalid" is true,

then the statement

All variables in the universe are related/influence each other/are affected by each other/etc. must be true.

Otherwise, penalization would not occur with variable selection.

Answer 4

I'm going to say yes. Of course something being an outlier on one measure does not necessarily mean it will be an outlier on other measures, but I think it certainly increases the odds. In essence, I think anything being unusual in one way is more likely to be unusual in other ways. This is perhaps the result of different variables being correlated in all kinds of ways. So if one is "extreme", this will also correlate with other factors that might also be in the extreme range.

I hope someone can provide a better answer with some empirical evidence. Perhaps we can look at this from the multivariate outlier perspective. In my experience, individuals who are outliers on one variable are often outliers on many other variables as well, contributing to multivariate outlierness.

Take this answer with a grain of salt since it depends on how we define an outlier and such, but I think in general it makes sense.

Answer 5

I will try to reply using empirical evidence. Let's assume you are measuring the heights in a men sample. In this case, the outlier will be represented by a very tall men (a giant). It is very likely this men will represent an outlier also for other variables like for instance shoe size or arms lengths and so on. Other case, you are measuring financial performance of US Public company. An outlier will be a very successful company with a sales growth twice the industry average. Very likely the same company will be an outlier in respect of any measure of profitability or stock price appreciation. In a nut shell, I am incline to think something behaving exceptionally out of the norm will tend to conserve this property across different manifestations. Is there a theorem that disprove this theory?

Answer 6

Outliers are a reflection of an unknown/unspecified external factor. If there is a relationship between two series then there would be an increased probability that both series would be affected. My answer to your question is "yes" since there may be a relationship between the two series.

Answer 7

The answer, in my mind, is "Yes and No, and if Yes then this isn't actually interesting".

For variables with no dependency structure, the answer is no - a very high, or very low, value of a particular variable doesn't imply a very high, or very low value of another variable.

For variables with a dependency structure, the answer is often (but not always) yes. But this isn't a property of "outlier-ness", it's a property of correlation itself. What you're showing is not that "Being an outlier begets being an outlier in other areas" but that two associated variables behave exactly like associated variables should.

Answer 8

If I were doing a binomial test, and I had 100% of my results indicating pass until the nth sample, at which time I had one sample indicate fail, then I would say that my estimate of the binomial distribution parameters and their confidence intervals might be very different before versus after the "fail".

My binomial test is whether or not the samples belong to an expected distribution which if true gets a "pass" or whether they qualify as an outlier in which case the result is "fail".

Keywords of interest may include "zero defect sampling", and "acceptance sampling".

Reference links:

• ASQ article
• Calculator