9

I am new to statistics. I am getting my hands dirty on VarianceThreshold. I am having a single dimensional array, containing N values. What's the maximum and minimum values of a variance for any values present in array?

I guess the minimum value will always be non-negative but I don't know about the maximum value. I have googled it but couldn't find a good answer.

mkt
  • 18,245
  • 11
  • 73
  • 172
Mangu Singh Rajpurohit
  • 219
  • 6
    There is no theoretical upper limit on the maximum variance of a sample. The minimum possible variance is zero of course. There are however practical limits to the biggest number which can be represented in a particular programming language/machine. – Daniel López Jun 07 '18 at 14:15
  • 4
    (Bit unfair that this gets downvoted so harshly.) On substance: it all depends on what you assume about the underlying probability distribution. Say, you assume a uniform distribution on a bounded interval, then clearly your variance will be bounded. On the other hand, if you assume a t-distribution with 2 degrees of freedom, the population variance will be infinite. – Jim Jun 07 '18 at 14:47
  • 2
    @Jim I agree that the downvote is a little harsh especially since the OP made clear that what is being asked about is the variance of the numbers in the array. This will always be finite even if the numbers in the array come from a $t$-distribution as long as we abide by the usual assumption that the numbers in the array are from the ordinary real numbers and not the extended real numbers that include $\pm\infty$. – Dilip Sarwate Jun 07 '18 at 15:13
  • 6
    What is "VarianceThreshold"? – Juho Kokkala Jun 07 '18 at 17:16
  • @DilipSarwate I expanded my comment into an answer. I believe it all hinges on the boundedness of support assumption that one is willing to make. I look forward to your thoughts on the matter. – Jim Jun 07 '18 at 21:01
  • It's unclear whether you're uncertain about either (1) what value the variance of a particular array might take because you only have limited information about it, or (2) what values the variances of arrays created by a particular process might take. If 1, could you explain what information do you have. If 2, could you give details of the process? – Scortchi - Reinstate Monica Jun 21 '18 at 08:00

2 Answers2

13

I interpret this question as asking

Given a set of $N$ numbers $x_1, x_2, \ldots,x_N$, what is the minimum and maximum values that the variance $V$, defined as $$V = \frac 1N \sum_{k=1}^N (x_k-\bar{x})^2 ~~ \text{where}~\bar{x}=\frac 1N \sum_{k=1}^N x_k$$ can take on?

Well, the minimum value of $V$ is $0$ as Daniel Lopez's comment points out, and it occurs if and only if all the $N$ numbers have the same value. At the other end, every finite set of real numbers has a (finite) upper bound (call it $b$) and a (finite) lower bound (call it $a$), and
$$V \leq \frac{(b-a)^2}{4} = \left(\frac{b-a}{2}\right)^2 = \left(\frac{\mathcal R}{2}\right)^2\tag{1}$$ where $\mathcal R$ is the range of the set of $N$ numbers. Note that it is not necessary to know the values of $b$ and $a$ separately; we only need the range $\mathcal R = (b-a)$ to calculate the upper bound $(1)$ on $V$.

If $N$ is an even number, there exist sets for which the bound $(1)$ holds with equality: these are sets for which $\frac N2$ of the $x_k$ have value $b$ and the other $\frac N2$ have value $a$. For odd $N$, the bound $\frac{(b-a)^2}{4}$ still applies but cannot be attained with equality if $N>1$. For odd $N>1$, the maximum value is $\frac{(b-a)^2}{4}\frac{N^2-1}{N^2}$ and is attained by a set in which $\frac{N-1}{2}$ of the $x_k$ have value $b$, $\frac{N+1}{2}$ have value $a$, or vice versa. For details, see this answer of mine on math.SE.

In another answer (and the comments on it), @Jim has argued that "A set of $N$ real numbers" tells the listener nothing whatsoever about the set if you don't know what any of the values are, even the minimum or maximum, and so the only completely correct answer is that maximum possible $V$ is unbounded: any other answer (such as mine above or a couple of possible answers suggested by Jim) must be be festooned with caveats that the answer is based on assumptions might be unwarranted. I disagree. Even if a secretive questioner is unwilling to share any details about the set he/she is concerned about, my answer gives the questioner enough information to find the maximum possible value of $V$ for him/herself from very minimal information abut the set: just the range suffices, no need to know even what $N$ is!

EDIT: (by AHK) Corrected the maximum variance for odd $N>1$ and the corresponding choice of $x_i$'s

blooraven
  • 429
Dilip Sarwate
  • 46,658
  • 2
    Please provide a correct summary of my answer and comments. To aide you: I consider $N$ known. More importantly, I discuss different cases of a priori knowledge concerning the population (this is what makes it a statistics question/answer). I hope this is clear now. If not I'd happily edit your answer to provide a correct and honest representation. – Jim Jun 16 '18 at 12:02
  • 1
    (+1) I don't really see any conflict with @Jim's answer - which deals mainly with what you can say about the values it's possible for a sample variance to take based on the data-generating process. (It's not at all clear what situation the OP's in.) – Scortchi - Reinstate Monica Jun 21 '18 at 07:46
3

It depends... But on what exactly?

The answer depends on the assumptions that you make.

1. If I read your question most literally: you know all data values.

In that case, there is no need for bounds (minimum or maximum), as you can simply calculate the variance of the data values in the array with:

$$ \text{var}(\mathbf{x}) = \frac{1}{N} \sum_{i=1}^N (x_i - \bar{x})^2 \, . $$

2. Now, say, you do not know any of the values; only that there are $N$. In other words: you have not seen the sample, but only know the sample size.

Then the answer depends on what you assume about where the sample came from, i.e. the population.

2.1 If you make no assumptions about the population (equivalently: the underlying distribution), then you cannot say anything about an upperbound for the sample variance.

Take the example of a $t$-distribution with $\nu \le 2$ degrees of freedom. The population variance is infinite, and the sample variance cannot be bounded from above (unless $N = 1$). Why? Because for every sample you provide, I can increase its variance by pushing the minimum and maximum values farther from the mean.

Please note, this same argument holds for a standard normal distribution! Even-though it has population variance equal to $1$, one can create samples with arbitrarily large sample variance.

2.2 If you assume that the population "lives on" bounded support, then Dilip Sarwate's answer will suffice: on support $[0, \, c]$ the sample variance is maximally $c^2 / 4$ (multiplied by $(N-1)/N$ for odd $N$).

P.S. Since the variance is essentially a weighted sum (integral) of non-negative terms (integrand), it is non-negative itself and bounded from below by $0$. I therefore concentrated on the upperbound in my answer.

Jim
  • 2,152
  • 1
    Re 2.1 and 2.2 Any collection of $N$ real numbers (it doesn't matter diddlysquat whether they are samples from some population or simply pulled out one's anatomy) has a finite lower bound $a$ and a finite upper bound $b$, and so the variance of these $N$ numbers is at most $\displaystyle \frac{(b-a)^2}{4}$ with the bound being attainable if $N$ is even. It doesn't matter in the least whether the underlying population lives on bounded support or not or whether the population variance is infinite or not, or for that matter even whether the population mean exists or not. (continued) – Dilip Sarwate Jun 11 '18 at 14:28
  • If you have $N$ numbers that are samples from a Cauchy distribution, these $N$ numbers do have a finite and well-defined sample mean and variance (sample variance if you prefer), and the variance is still bounded by $\displaystyle \frac{(\max - \min)^2}{4}$ even though the population has undefined mean and so one cannot even begin to define the population variance. – Dilip Sarwate Jun 11 '18 at 14:35
  • @DilipSarwate On substance. 1) In your own answer you explicitly assume bounded support $[0,, c]$ (of the underlying distribution)! Therefore, it is a bit hard to follow your argument in the comments. 2) It seems to me, that you may be confusing the sample after observing it – case 1. – with the a priori cases of 2.. 3) To drive my point home, consider the (ordered) samples: $S_1 = { X_{(1)} \le X_{(2)} \le ...< ... \le X_{(N)} }$ and $S_2 = { X_{(1)} -1 \le X_{(2)} \le ...< ... \le X_{(N)}+1 }$. Then $var(S_1) < var(S_2)$; continue this construction ad infinitum... – Jim Jun 15 '18 at 14:32
  • @DilipSarwate On form. Please be kind on this forum and mind your language. You may learn something. – Jim Jun 15 '18 at 14:38
  • The question is about the maximum and minimum possible values that the variance of a set of $N$ numbers can take on; where those numbers came from is irrelevant. That set of real numbers has minimum variance $0$ when all numbers are equal and maximum variance $\frac{(\max-\min)^2}{4}$ etc. That $var(S_1)<var(S_2)$ has nothing to do with it. – Dilip Sarwate Jun 15 '18 at 14:42
  • 1
    @DilipSarwate you are assuming that you know the (sample) extrema! You would know these after observing the sample. But if you know all values in the array, why bound the variance in the first place? You can easily calculate its exact value! No need for bounds. Please note, that the question is not: "Given sample size $N$ and the extrema, what are the bounds that the variance can take?" – Jim Jun 15 '18 at 14:55
  • There is no need to argue about what is the question. When the question is unclear because of the OP being new to statistics, it's better to (educatedly) guess what he needs to know and give an answer, or as many answers as reasonable guesses can be drawn. – Pere Jun 16 '18 at 07:13
  • @Pere Thank you, but that is what the comments are for: clarifying what the question is exactly. Please note that comments by Daniel López and me preceded Dilip Sarwate's answer. Note further that in the Math.SE question the OP clearly defines bounded support $[0,,p]$. Finally, the considerations in my answer are what makes this statistics versus purely mathematical. – Jim Jun 16 '18 at 11:50
  • (+1) I don't really see any conflict with @DilipSarwate's answer - which just considers another case in between (1) & (2) in which you don't know all data values but know a statistic, the range. (It's not at all clear what situation the OP's in.) – Scortchi - Reinstate Monica Jun 21 '18 at 07:45