7

I was reading about weight initialization in neural networks (He et. al, 2015) when I came across this statement:

"If we let $w_{l-1}$ have a symmetric distribution around zero and $b_{l} = 0$, then $y_{l-1}$ has zero mean and a symmetric distribution around zero."

Where $y_{l-1}$ is given by $w_{l-1}x_{l-1}$, the product of the layer's weights and its inputs. Here $w_{l-1}$ is generated by a normal distribution with mean zero, so it is symmetric. I know how to prove that the mean of this product is zero, but how would I go about showing it is also symmetric? In other words, given two random variables $X$ and $Y$ with $X$ being symmetric, how can I show $XY$ is also symmetric?

whuber
  • 322,774
BlackKnight
  • 173
  • 3
  • 1
    Apply any definition you like of "symmetric." Do you have a particular one in mind? – whuber Mar 01 '23 at 20:44
  • Symmetric meaning p(E[X] + k) = p(E[X] - k) for all k – BlackKnight Mar 01 '23 at 20:52
  • 2
    Be careful! (1) Symmetric distributions don't necessarily have expectations. For a rigorous definition see https://stats.stackexchange.com/questions/493728 or https://stats.stackexchange.com/questions/28992. (2) If both expectations are nonzero, the conclusion is incorrect. For an example, let $X$ give equal probabilities to $0$ and $2$ and $Y$ give equal probabilities to $1$ and $3.$ Both are symmetric, but their product gives a probability of $1/2$ to $0$ and probabilities of $1/4$ to $2$ and $6,$ which is obviously not symmetric. – whuber Mar 01 '23 at 20:55
  • 2
    You will definitely need to specify the distribution of $x_{l - 1}$ (and probably how it interacts with $w_{l - 1}$) to reach the conclusion that $y_{l - 1}$ is symmetric around zero, because the statement "given two random variables $X$ and $Y$ with $X$ being symmetric, then $XY$ is also symmetric." is obviously wrong. For a simple counterexample, take $X = Y \sim N(0, 1)$. Is $X^2$ symmetric? – Zhanxiong Mar 01 '23 at 21:18

1 Answers1

15

To say that a random variable $W$ "has a symmetric distribution around zero" is saying that $W$ and $-W$ have the same distribution.

Let $X$ be another random variable and set $Y=WX.$ By the rules of algebra, $-Y = -WX = (-W)X$ must have the same distribution as $WX$ provided $W$ and $X$ are independent. Therefore $Y$ is symmetric around zero.

To see why independence is a necessary assumption, let $(W,X)$ take on the values $(-1,0),$ $(-1,1),$ $(1,0),$ and $(1,1)$ with probabilities $1/3,1/6,1/6,$ and $1/3,$ respectively. Because $W$ has equal chance of being $\pm 1,$ it is symmetric around zero. ($X$ is symmetric about $1/2.$) But $Y=WX$ takes on the value $0$ with probability $1/3+1/6,$ the value $-1$ with probability $1/6,$ and the value $1$ with probability $1/3,$ and therefore is not symmetric at all.

$$\begin{array}{crrr} & \Pr & W & X & Y \\ \hline & \frac{1}{3} & -1 & 0 & 0\\ & \frac{1}{6} & -1 & 1 & -1\\ & \frac{1}{6} & 1 & 0 & 0\\ & \frac{1}{3} & 1 & 1 & 1 \end{array}$$

whuber
  • 322,774