Does sample mean always has normal distribution $\mathcal{N}(a, \frac{\sigma^2}{n})?$

Question

The question was inspired by these comments to my other question. I have proved this proposition some time ago and could not find any issues in it.

It consists of two parts:

1. Where does my proof of $\overline{\xi_n}=\frac{\xi_1+...+\xi_n}{n}\xrightarrow{d} \mathcal{N}(a, \frac{\sigma^2}{n})$ break down?

2. Why (if this proposition is incorrect) this fact is being widely used in lots of lecture notes which I found before and how one can correctly interpret corollaries from CLT? I guess I will ask this question as another post after I will understand the first part.

First part.

I will assume that we know from CLT that If $\xi_1,...,\xi_n$ are identically distributed independent random variables with $\mathbb{E}\xi_1=a, \mathbb{V}ar\xi_1=\sigma^2$ then

$\frac{\xi_1+...+\xi_n - na}{\sqrt{n\sigma^2}} \xrightarrow{d} \eta,$ where $\eta \sim \mathcal{N}(0,1)$, so by definition of convergence in distribution,

$\forall x \in \mathbb{R}$, $\mathbb{P}[\frac{\xi_1+...+\xi_n - na}{\sqrt{n\sigma^2}}\leq x] \xrightarrow{n\to+\infty} \mathbb{P}[\eta \leq x].$

So we have $\mathbb{P}[\overline{\xi_n}\leq x]=\mathbb{P}[\frac{\sigma}{\sqrt{n}}\frac{\xi_1+...+\xi_n-na}{\sqrt{n\sigma^2}}+a \leq x] = \mathbb{P}[\frac{\xi_1+...+\xi_n - na}{\sqrt{n\sigma^2}}\leq\frac{x-a}{\sigma}\sqrt{n}]$.

But then, $\mathbb{P}[\frac{\xi_1+...+\xi_n - na}{\sqrt{n\sigma^2}}\leq\frac{x-a}{\sigma}\sqrt{n}] \xrightarrow{n\to+\infty}\mathbb{P}[\eta \le \frac{x-a}{\sigma}\sqrt{n}] = \mathbb{P}[\frac{\sigma\eta}{\sqrt{n}}+a\leq x]$

But we know if $\eta \sim \mathcal{N}(0,1),$ then $\frac{\sigma\eta}{\sqrt{n}}+a$ has distribution $\zeta \sim$ $\mathcal{N}(a, \frac{\sigma^2}{n})$.

This means that $\mathbb{P}[\overline{\xi_n}\leq x] \xrightarrow{n\to+\infty}$ $\mathbb{P[\zeta\leq x]}$, (I denote this proposition as *) and by definition of convergence in distribution,

$\overline{\xi_n}=\frac{\xi_1+...+\xi_n}{n}\xrightarrow{d} \mathcal{N}(a, \frac{\sigma^2}{n})$ (I denote this proposition by **).

Answer 1

Background: This answer is prompted by the discussion between the OP and myself in the comments on the OP's question.

The title of the OP's question asks

Does sample mean always has normal distribution?

to which the answer is a resounding NO. The sample mean of $n$ i.i.d random variables $X_1, X_2, \ldots, X_n$ with finite mean $\mu$ and finite variance $\sigma^2$ almost never has a normal distribution, with the sole exception that requires the use of "almost" is the case when the $X_k$'s themselves are i.i.d normal random variables!!

Turning to the OP's multiple misconceptions about what the Central Limit Theorem (CLT) actually says, the CLT is not a result about the sample mean $S_n =\displaystyle \frac 1n \sum_{k=1}^n X_k$ at all, but about a related quantity $\displaystyle\frac{1}{\sqrt{n}} \sum_{k=1}^n (X_k-\mu) = \sqrt{n} (S_n-\mu)$. Note that $S_n$ is a random variable with mean $\mu$ and variance $\displaystyle \frac{\sigma^2}{n}$ and thus $\sqrt{n} (S_n-\mu)$ is a zero-mean random variable with variance $\sigma^2$. The simplest form of the CLT claims that for large values of $n$, $P(\sqrt{n} (S_n-\mu)\leq x)$ has value approximately $\Phi\left(\dfrac{x}{\sigma}\right)$ (where $\Phi(\cdot)$ is the CDF of a standard normal random variable), and that the approximation gets better as $n$ increases. In fact, the limit as $n \to \infty$ of $P(\sqrt{n} (S_n-\mu)\leq x)$ is $\Phi\left(\dfrac{x}{\sigma}\right)$.

But the OP doesn't care about the random variable $\sqrt{n} (S_n-\mu)$ at all; the OP wants to study the asymptotic behavior of $P(S_n \leq x)$, and so let's do just that. We have \begin{align} P(S_n \leq x)&= P(S_n-\mu \leq x-\mu)\\ &= P((\sqrt{n}(S_n-\mu) \leq \sqrt{n}(x-\mu))\\ &\approx \Phi\left(\dfrac{\sqrt{n}(x-\mu)}{\sigma}\right)\\ &= \Phi\left(\dfrac{x-\mu}{\frac{\sigma}{\sqrt{n}}}\right). \end{align}

So, it is appropriate to aver that for large $n$, the CDF of the sample mean $S_n$ is well-approximated by the CDF of a $\mathcal N\left(\mu, \dfrac{\sigma^2}{n}\right)$ random variable. What is not correct to conclude is what the OP has been insisting on all along in the face of numerous attempts by many people trying to correct his misconception: that in the limit as $n \to \infty$, the CDF of $S_n$ is converging to the CDF of a $\mathcal N\left(\mu, \dfrac{\sigma^2}{n}\right)$ random variable. Notice that

• If $x<\mu$, then $$\lim_{n\to\infty}P(S_n \leq x) = \lim_{n\to\infty} \Phi\left(\dfrac{\sqrt{n}(x-\mu)}{\sigma}\right)$$ where the argument of $\Phi(\cdot)$ is diverging to $-\infty$ as $n \to -\infty$ and so for all $x <\mu$, $\lim_{n\to\infty}P(S_n \leq x)=0.$

• If $x>\mu$, then $$\lim_{n\to\infty}P(S_n \leq x) = \lim_{n\to\infty} \Phi\left(\dfrac{\sqrt{n}(x-\mu)}{\sigma}\right)$$ where the argument of $\Phi(\cdot)$ is diverging to $\infty$ as $n \to \infty$ and so for all $x > \mu$, $\lim_{n\to\infty}P(S_n \leq x)=1$.

We conclude that in the limit as $n\to \infty$, the CDF of $S_n$ has value $0$ for $x < \mu$, and value $1$ for $x>\mu$.

The limit distribution of $S_n$ is the constant $\mu$ and not a normal distribution at all, except insofar as the constant $\mu$ can be regarded as a degenerate normal random variable with mean $\mu$ and variance $0$.

Answer 2

For a fixed $x$, statements of the form $F_{X_n}(x) \overset{n \to \infty}\longrightarrow F_X(g_n(x))$ or $F_{X_n}(x) \overset{n \to \infty}\longrightarrow F_{h_n(X)}(x)$, where $g_n$ and $h_n$ depend on $n$, do not adhere to the concept of pointwise convergence (excluding points of discontinuity) of a sequence of CDFs and thus don't describe convergence in distribution.

---

If $a$ and $\sigma^2$ are finite, the Lindeberg-Lévy CLT gives $$ \sqrt{n}\left(\bar{\xi}_n - a\right) \overset{d}{\underset{n \to \infty}\longrightarrow} \mathop{\mathcal N}\left(0, \sigma^2\right), $$ where $\bar{\xi}_n = \frac{1}{n}\sum_{i=1}^n\xi_i$.

To show that this implies that $\bar{\xi}_n$ converges in distribution (or, equivalently, in probability) to $a$ as $n$ tends to infinity we can rewrite $\bar{\xi}_n$ as follows $$ \bar{\xi}_n = \underbrace{\frac{1}{\sqrt{n}}}_{{\underset{n \to \infty}\longrightarrow}0} \underbrace{\sqrt{n}\left(\bar{\xi}_n - a\right)}_{\overset{d}{\underset{n \to \infty}\longrightarrow} \mathop{\mathcal N}\left(0, \sigma^2\right)} + a. $$ Now, applying Slutsky's theorem, we have $$ \bar{\xi}_n \overset{d}{\underset{n \to \infty}\longrightarrow} 0 + a = a \iff \bar{\xi}_n \overset{p}{\underset{n \to \infty}\longrightarrow} a. $$

What is often used as an approximation is to act as if

$$ \sqrt{n}\left(\bar{\xi}_n - a\right) \sim \mathop{\mathcal N}\left(0, \sigma^2\right) $$

would hold for a finite sample size $n$. Under that assumption $$ \bar{\xi}_n \sim \mathop{\mathcal N}\left(a, \frac{\sigma^2}{n}\right) $$ is true and can be understood as an approximation to the finite-sample distribution of $\bar{\xi}_n$.