I am puzzled by the following statement:
" In order to increase the standard deviation of a set of numbers, you must add a value that is more than one standard deviation away from the mean"
What is the proof of that? I know of course how we define the standard deviation but that part I seem to miss somehow. Any comments?
For any $N$ numbers $y_1,y_2, \ldots, y_N$ with mean $\displaystyle \bar{y} = \frac{1}{N}\sum_{i=1}^N y_i$, the variance is given by $$\begin{align} \sigma^2 &= \frac{1}{N-1}\sum_{i=1}^N (y_i-\bar{y})^2\\ &= \frac{1}{N-1}\sum_{i=1}^N \left(y_i^2 - 2y_i\bar{y} + \bar{y}^2\right)\\ &= \frac{1}{N-1}\left[\left(\sum_{i=1}^Ny_i^2\right) - 2N(\bar{y})^2 + N(\bar{y})^2 \right] \\ \sigma^2 &=\frac{1}{N-1}\sum_{i=1}^N \left(y_i^2 - (\bar{y})^2\right) \tag{1} \end{align}$$ Applying $(1)$ to the given set of $n$ numbers $x_1, x_2, \ldots x_n$ which we take for convenience in exposition to have mean $\bar{x} = 0$, we have that $$\sigma^2 = \frac{1}{n-1}\sum_{i=1}^n \left(x_i^2-(\bar{x})^2\right) = \frac{1}{n-1}\sum_{i=1}^n x_i^2$$ If we now add in a new observation $x_{n+1}$ to this data set, then the new mean of the data set is $$\frac{1}{n+1}\sum_{i=1}^{n+1}x_i = \frac{n\bar{x} + x_{n+1}}{n+1} = \frac{x_{n+1}}{n+1}$$ while the new variance is $$\begin{align} \hat{\sigma}^2 &= \frac{1}{n}\sum_{i=1}^{n+1} \left(x_i^2-\frac{x_{n+1}^2}{(n+1)^2}\right)\\ &= \frac{1}{n}\left[\left((n-1)\sigma^2 + x_{n+1}^2\right) - \frac{x_{n+1}^2}{n+1}\right]\\ &= \left.\left.\frac{1}{n}\right[(n-1)\sigma^2 + \frac{n}{n+1}x_{n+1}^2\right]\\ &> \sigma^2 ~ \text{only if}~ x_{n+1}^2 > \frac{n+1}{n}\sigma^2. \end{align}$$ So $|x_{n+1}|$ needs to be larger than $\displaystyle\sigma\sqrt{1+\frac{1}{n}}$ or, more generally, $x_{n+1}$ needs to differ from the mean $\bar{x}$ of the original data set by more than $\displaystyle\sigma\sqrt{1+\frac{1}{n}}$, in order for the augmented data set to have larger variance than the original data set. See also Ray Koopman's answer which points out that the new variance is larger than, equal to, or smaller than, the original variance according as $x_{n+1}$ differs from the mean by more than, exactly, or less than $\displaystyle\sigma\sqrt{1+\frac{1}{n}}$.
The puzzling statement gives a necessary but insufficient condition for the standard deviation to increase. If the old sample size is $n$, the old mean is $m$, the old standard deviation is $s$, and a new point $x$ is added to the data, then the new standard deviation will be less than, equal to, or greater than $s$ according as $|x-m|$ is less than, equal to, or greater than $s\sqrt{1+1/n}$.
Leaving aside the algebra (which also works) think about it this way: The standard deviation is square root of the variance. The variance is the average of the squared distances from the mean. If we add a value that is closer to the mean than this, the variance will shrink. If we add a value that is farther from the mean than this, it will grow.
This is true of any average of values that are non-negative. If you add a value that is higher than the mean, the mean increases. If you add a value that is less, it decreases.
I'll get you started on the algebra, but won't take it quite all of the way. First, standardize your data by subtracting the mean and dividing by the standard deviation: $$ Z = \frac{x-\mu}{\sigma} .$$ Note that if $x$ is within one standard deviation of the mean, $Z$ is between -1 and 1. Z would be 1 if $x$ were exactly one sd away from the mean. Then look at your equation for standard deviation: $$\sigma = \sqrt{\frac{\sum_{i=1}^{N}Z_i^2}{N-1}}$$ What happens to $\sigma$ if $Z_N$ is between -1 and 1?
