2

Referring to: How to understand that MLE of variance is biased in a Gaussian distribution

at some point during calculation the formula of the sum of the expected value becomes a single expected value:

confusing formula

The explanation given is: With the last step following since due to $E[X^2_n]$ being equal across n due to coming from the same distribution.

I can't understand:

  1. What does the expectation of a value mean, in this case what is even $E[X_n]$
  2. Why should the expected value of an observation value be equal?
  3. What's the proof for $Var[1/N *...] = (1/N)^2 Var[...]$

Thanks for the help and sorry for the bad format I don't know how to write formulas, this is my first post xD

gunes
  • 57,205
Cristian
  • 33

2 Answers2

1

Note that $x_n$'s are random variables, not specific numbers. Assuming that $x_n$ is gaussian distributed with density $p$, it's expectation is per definition $$ E[x_n] = \int_\mathbb{R} x \cdot p(x) dx. $$ If we apply a function to $x_n$, it's expectation is $$E[f(x_n)] = \int_\mathbb{R} f(x) \cdot p(x) dx, $$ in your case $f(x) = x^2$.

The calculation you reference, assumes that the $x_n$ all come from the same distribution, i.e. they have the same density $p$ and thus the same expectation.

For your last question see e.g. here for a more general statement and proof.

blat
  • 156
  • According to book though the Xn are the values of one observation, as stated: "we obtain the maximum likelihood solution which is the sample mean, i.e., the mean of the observed values {xn}"

    and

    "We first note that the maximum likelihood solutions µML and σ2ML are functions of the data set values x1,...,xN. Consider the expectations of these quantities with respect to the data set value"

     – Cristian Feb 10 '20 at 13:05
  • If this is the case the expected value shouldn't be different for each observation? – Cristian Feb 10 '20 at 13:10
  • 1
    @Cristian It's not different for each observation because they're sampled from the same dist. – gunes Feb 10 '20 at 13:16
1

$X_n$ is a Gaussian RV for each $n$; since iid, they all have the same mean, variance, moments etc. i.e. simply we have $$E[X_1^2]=E[X_2^2]=\dots=E[X_n^2]$$

Therefore, $\frac{1}{N}\sum_{n=1}^N E[X_n^2]=\frac{1}{N}NE[X_1^2]=E[X_2^2]=E[X_n^2]$ for any $n$. This is why we can remove the summation.

The proof for scaling property of variance can be: $$\operatorname{var(cX)}=E[(cX)^2]-E[cX]^2=c^2(E[X^2]-E[X]^2)=c^2\operatorname{var}(X)$$

gunes
  • 57,205
  • According to book though the Xn are the values of one observation, as stated: "we obtain the maximum likelihood solution which is the sample mean, i.e., the mean of the observed values {xn}"

    and

    "We first note that the maximum likelihood solutions µML and σ2ML are functions of the data set values x1,...,xN. Consider the expectations of these quantities with respect to the data set value". If this is the case the expected value shouldn't be different for each observation?

     – Cristian Feb 10 '20 at 13:07
  • @Cristian This is the chicken & egg problem most novice practitioners are trapped in. $X_n$'s are observations sampled from a distribution. So, they're actually random variables o/w we wouldn't have sampled them and they wouldn't belong to a distribution. They're not fixed constants. Knowing (or pretending to know) and using them in our calculations doesn't mean that they're not RVs. In ML estimators, you pretend if you know $X_n$, use them in your estimator formulas, but when it comes to evaluating the bias of your estimator, we just take the expectation. How hypocrite of us :) – gunes Feb 10 '20 at 13:14
  • Yeah I'm quite new to random processes and this is really confusing for me, so you say that even tough the values come from an observation we consider them as variables? How would the expectation formula change? would we have a constant in the integration or what? – Cristian Feb 10 '20 at 13:20
  • In any expected value calculation, you don't deal with $x_n,X_n$ or etc. When you know the PDF, e.g. $x_n\sim f(x)$, the expected value can be written with any variable: $$E[x_n]=\int \tau f(\tau)d\tau$$ – gunes Feb 10 '20 at 13:25
  • Ok, so basically you don't care at all about the value, you say this is a value given by a certain pdf, so it's expected value is given by the formula which is the same for all values since they have the same PDF, that feels so wrong though! Is that the case in general? As far as I understood the expected value referred to a distribution, so I was wrong on that since it's about a variable that comes from that distribution? Also when we multiply Xn do we multiply the values or the distributions? – Cristian Feb 10 '20 at 13:37
  • While calculating statistics from your data sample, you use the values numerically, but the expected value is like asking if regardless of the current value, what would be the average value I'd get if I draw a sample. Consider a population, and we sample 10 people and measure their heights: name them as $X_1,..,X_{10}$ belonging to John, Jack etc. But, $X_1$ is not always John, it is the first person you choose from the population. So, it's going to change if you sample tomorrow, which means $X_1$ is a RV, but John is not. So, in expected value, you care about $X_1$, not John. – gunes Feb 10 '20 at 13:44
  • 1
    Wonderful I think I get it, so it's the same also for the variance for example, technically we could evaluate the variance of a single value (coming from a sample) but since we don't know the pdf we need multiple observation to attempt to determine it, right? – Cristian Feb 10 '20 at 13:52
  • yes, variance is also an expected value: $$var(X)=E[(X-E[X])^2]$$ a note: variance of a known value is $0$ since it doesn't vary, e.g. $var(3)=0$. But, variance of a RV is calculated through the distribution. Same idea applies. – gunes Feb 10 '20 at 14:00