Independence of Mean and Variance of Discrete Uniform Distributions

Question

In the comments below a post of mine, Glen_b and I were discussing how discrete distributions necessarily have dependent mean and variance.

For a normal distribution it makes sense. If I tell you $\bar{x}$, you haven't a clue what $s^2$ is, and if I tell you $s^2$, you haven't a clue what $\bar{x}$ is. (Edited to address the sample statistics, not the population parameters.)

But then for a discrete uniform distribution, doesn't the same logic apply? If I estimate the center of the endpoints, I don't know the scale, and if I estimate the scale, I don't know the center.

What's going wrong with my thinking?

EDIT

I did jbowman's simulation. Then I hit it with the probability integral transform (I think) to examine the relationship without any influence from the marginal distributions (isolation of the copula).

Data.mean <- Data.var <- rep(NA,20000)
for (i in 1:20000){     
    Data <- sample(seq(1,10,1),100,replace=T)
    Data.mean[i] <- mean(Data)
    Data.var[i] <- var(Data)    
}
par(mfrow=c(2,1))
plot(Data.mean,Data.var,main="Observations")
plot(ecdf(Data.mean)(Data.mean),ecdf(Data.var)(Data.var),main="'Copula'")

In the little image that appears in RStudio, the second plot looks like it has uniform coverage over the unit square, so independence. Upon zooming in, there are distinct vertical bands. I think this has to do with the discreteness and that I shouldn't read into it. I then tried it for a continuous uniform distribution on $(0,10)$.

Data.mean <- Data.var <- rep(NA,20000)
for (i in 1:20000){

    Data <- runif(100,0,10)
    Data.mean[i] <- mean(Data)
    Data.var[i] <- var(Data)

}
par(mfrow=c(2,1))
plot(Data.mean,Data.var)
plot(ecdf(Data.mean)(Data.mean),ecdf(Data.var)(Data.var))

This one really does look like it has points distributed uniformly across the unit square, so I remain skeptical that $\bar{x}$ and $s^2$ are independent.

That's an interesting approach you've taken there, I'll have to think about it. — jbowman, Aug 18 '19 at 17:12
The dependence (necessarily) gets weaker at larger sample sizes so it's hard to see. Try smaller sample sizes, like n=5,6,7 and you'll see it more easily. — Glen_b, Aug 19 '19 at 00:09
@Glen_b You're right. There's a more obvious relationship when I shrink down the sample size. Even in the image I posted, there appears to be some clustering in the lower right and left corners, which is present in the plot for the smaller sample size. Two follow-ups. 1) Is the dependence necessarily getting weaker because the population parameters can be varied independent of each other? 2) It seems wrong that the statistics would have any kind of dependence, but they clearly do. What causes this? — Dave, Aug 19 '19 at 00:30
One way to get some insight is to examine the special features of the samples that get into those 'horns" at the top of Bruce's plots. In particular note that at n=5, you get the largest possible variance by all the points being close to 0 or 1, but because there's 5 observations, you need 3 at one end and 2 at the other, so the mean must be near to 0.4 or 0.6 but not near 0.5 (since putting one point in the middle will drop the variance a bit). If you had a heavy tailed distribution, both mean and variance would be most impacted by the most extreme observation ... ctd — Glen_b, Aug 19 '19 at 00:40
ctd... and in that situation you get a strong correlation between $|\bar{x}-\mu|$ and $s$ (giving two big "horns" either side of the population center on a plot of sd vs mean) -- with the uniform this correlation is somewhat negative. ... With large samples you'll head toward the asymptotic behavior of $(\bar{X},s^2_X)$ which ends up being jointly normal. — Glen_b, Aug 19 '19 at 00:42

score 6 · Answer 1 · edited Dec 10 '21 at 00:35

6

jbowman's Answer (+1) tells much of the story. Here is a little more.

(a) For data from a continuous uniform distribution, the sample mean and SD are uncorrelated, but not independent. The 'outlines' of the plot emphasize the dependence. Among continuous distributions, independence holds only for normal.

    set.seed(1234)
    m = 10^5; n = 5
    x = runif(m*n);  DAT = matrix(x, nrow=m)
    a = rowMeans(DAT)
    s = apply(DAT, 1, sd)
    plot(a,s, pch=".")

(b) Discrete uniform. Discreteness makes it possible to find a value $a$ of the mean and a value $s$ of the SD such that $P(\bar X = a) > 0,\, P(S = s) > 0,$ but $P(\bar X = a, X = s) = 0.$

    set.seed(2019)
    m = 20000;  n = 5;  x = sample(1:5, m*n, rep=T)
    DAT = matrix(x, nrow=m)
    a = rowMeans(DAT)
    s = apply(DAT, 1, sd)
    plot(a,s, pch=20)

(c) A rounded normal distribution is not normal. Discreteness causes dependence.

    set.seed(1776)
    m = 10^5; n = 5
    x = round(rnorm(m*n, 10, 1));  DAT = matrix(x, nrow=m)
    a = rowMeans(DAT);  s = apply(DAT, 1, sd)
    plot(a,s, pch=20)

(d) Further to (a), using the distribution $\mathsf{Beta}(.1,.1),$ instead of $\mathsf{Beta}(1,1) \equiv \mathsf{Unif}(0,1).$ emphasizes the boundaries of the possible values of the sample mean and SD. We are 'squashing' a 5-dimensional hypercube onto 2-space. Images of some hyper-edges are clear. [Ref: The figure below is similar to Fig. 4.6 in Suess & Trumbo (2010), Intro to probability simulation and Gibbs sampling with R, Springer.]

    set.seed(1066)
    m = 10^5; n = 5
    x = rbeta(m*n, .1, .1);  DAT = matrix(x, nrow=m)
    a = rowMeans(DAT);  s = apply(DAT, 1, sd)
    plot(a,s, pch=".")

Addendum per Comment.

edited Dec 10 '21 at 00:35

kjetil b halvorsen

77,844

answered Aug 18 '19 at 17:32

BruceET

56,185

Use ecdf on your last one. The scatterplot is wild! Anyway, if a uniform variable has dependence between $\bar{x}$ and $s^2$, how is it that we're getting some information about one by knowing the other, given that we can stretch the range or shift the center willy nilly and not affect the other value? If we get $\bar{x}=0$, we shouldn't know if $s^2 = 1$ or $s^2=100$, similar to how we can stretch the normal distribution without affecting the mean. – Dave Aug 18 '19 at 19:49
1

The criterion of independence is demanding. Lack of independence btw two RVs does not guarantee that it's easy to get info about one, knowing the value of the other. // In (d), not sure what ECDF of A or S would reveal. // Scatterplot in (d) shows 6 'points', images under transformation of 32 vertices of 5-d hypercube with multiplicities 1, 5, 10, 10, 5, 1 (from left to right). Multiplicities explain why 'top two' points are most distinct. – BruceET Aug 18 '19 at 20:41
I don't mean that it's easy to get info about one if you know the other, but if you have independence, all you can go by is the marginal distribution. Consider two standard normal variables $X$ and $Y$ with $\rho=0.9$. If you know that $x=1$, you don't know what $y$ equals, but you know that a value around $1$ is more likely than a value around $-1$. If $\rho=0$, then a value around $1$ is just as likely as a value around $-1$. – Dave Aug 18 '19 at 20:52
But that's for a nearly-linear relationship btw two standard normals. Mean and SD of samples are not so easy. – BruceET Aug 18 '19 at 20:59
Even in your example D, $s=0.2$ is fairly likely as is $a=0.5$, but if you know that $a=0.5$, you would not be guessing that $s=0.2$. The dependence structure is more complicated than linear correlation, sure, but you still get to know something about $s$ by knowing $a$. – Dave Aug 18 '19 at 21:05
1

@Dave you do have information about one when you know the other. For example if the sample variance is really large, you know the sample mean isn't really close to 0.5 (see the gap at the top-centre of the first plot, for example) – Glen_b Aug 19 '19 at 00:12
@Glen_b Right, you have information about the other one, therefore dependence. (Though I do think we need to hit the variables with the probability integral transform before we graph them, so plot(ecdf(a)(a),ecdf(s)(s)), so we get rid of the influence of the marginal distributions.) – Dave Aug 19 '19 at 00:37
Sure, if you like; that won't change the existence of dependence; if you prefer to look at that way, go ahead, it shouldn't generally be a problem. My view is I prefer to use whatever makes the dependence most clear. In short tailed cases I often look at them as is or in some other cases transform to a convenient marginal distribution (e.g. to remove heavy skew) but sometimes the absolute deviation of the sample mean from the population mean is a good choice for plotting against $s$. – Glen_b Aug 19 '19 at 01:25
@Glen_b Hit it with the probability integral transform if I want to, you mean? I think that’s essential. Otherwise we could get misled by having something that doesn’t look like random scatter (so apparent dependence) but that is completely caused by the marginal distributions. Scrambling the “s” distribution in Bruce’s example D shows this. It looks like there’s dependence, but then the transform shows otherwise (and KDEs of the marginals are totally consistent with the pattern). – Dave Aug 19 '19 at 10:13
Example D demonstrably shows dependence. If you truly don't see it in the copula then it's obscuring dependence that's definitely there. – Glen_b Aug 19 '19 at 14:37

score 3 · Answer 2 · answered Aug 18 '19 at 16:24

It isn't that the mean and variance are dependent in the case of discrete distributions, it's that the sample mean and variance are dependent given the parameters of the distribution. The mean and variance themselves are fixed functions of the parameters of the distribution, and concepts such as "independence" don't apply to them. Consequently, you are asking the wrong hypothetical questions of yourself.

In the case of the discrete uniform distribution, plotting the results of 20,000 $(\bar{x}, s^2)$ pairs calculated from samples of 100 uniform $(1, 2, \dots, 10)$ variates results in:

which shows pretty clearly that they aren't independent; the higher values of $s^2$ are located disproportionately towards the center of the range of $\bar{x}$. (They are, however, uncorrelated; a simple symmetry argument should convince us of that.)

Of course, an example cannot prove Glen's conjecture in the post you linked to that no discrete distribution exists with independent sample means and variances!

That's a good catch about statistic versus parameter. I've made a pretty extensive edit. — Dave, Aug 18 '19 at 16:49

Independence of Mean and Variance of Discrete Uniform Distributions

2 Answers2

Linked