How to compare joint distribution to product of marginal distributions?

Question

I have two finite-sampled signals, $x_1$ and $x_2$, and I want to check for statistical independence.

I know that for two statistically independent signals, their joint probability distribution is a product of the two marginal distributions.

I have been advised to use histograms in order to approximate the distributions. Here's a small example.

x1 = rand(1, 50);
x2 = randn(1, 50);
n1 = hist(x1);
n2 = hist(x2);
n3 = hist3([x1' x2']);

Since I am using the default number of bins, n1 and n2 are 10-element vectors, and n3 is a 10x10 matrix.

My question is this: How do I check whether n3 is in fact a product of n1 and n2?

Do I use an outer product? And if I do, should I use x1'*x2 or x1*x2'? And why?

Also, I have noticed that hist returns the number of elements (frequency) of elements in each bin? Should this be normalized in any way? (I haven't exactly understood how hist3 works either..)

Thank you very much for your help. I'm really new to statistics so some explanatory answers would really help.

Answer 1

Assuming that the theoretical distributions of $x_1$ and $x_2$ are not known, a naive algorithm for determining independence would be as follows:

Define $x_{1,2}$ to be the set of all co-occurences of values from $x_1$ and $x_2$. For example, if $x_1 = { 1, 2, 2 }$ and $x_2 = { 3, 6, 5}$, the set of co-occurences would be $\{(1,3), (1, 6), (1, 5) , (2, 3), (2,6), (2,5), (2, 3), (2,6), (2,5))\}$.

1. Estimate the probability density functions (PDF's) of $x_1$, $x_2$ and $x_{1,2}$, denoted as $P_{x_1}$, $P_{x_2}$ and $P_{x_{1,2}}$.
2. Compute the mean-square error $y=sqrt(sum(P_{x_{1,2}}(y_1,y_2) - P_{x_1}(y_1) * P_{x_2}(y_2))^2)$, where $(y_1,y_2)$ takes the values of each pair in $x_{1,2}$.
3. if $y$ is close to zero, it means that $x_1$ and $x_2$ are independent.

A simple way to estimate a PDF from a sample is to compute the sample's histogram and then to normalize it so that the integral of the PDF sums to 1. Practically, that means that you have to divide the bin counts of the histogram by the factor $h * sum(n)$ where $h$ is the bin width and $n$ is the histogram vector.

Note that step 3 of this algorithm requires the user to specify a threshold for deciding whether the signals are independent.

Answer 2

If you are trying to do a test of independence, it's better to use well developed statistics than to come up with a new one. For example, you can start by computing the Chi-squared test. Of course, visualizing the difference between the product of marginals and the joint will give you a good insight, so I encourage you to compute it as well.

Answer 3

You could compare the joint empirical distribution function with the product of the marginal empirical distribution functions. For two samples $x=(x_1,\dots,x_{n_1})$ and $y=(y_1,\dots,y_{n_2})$, let $n=n_1+n_2$ and define the joint empirical distribution function $$ \hat{F}_n(s,t) = \frac{1}{n} \sum_{i=1}^{n_1} \sum_{j=1}^{n_2} I_{[x_i,\infty)\times[y_j,\infty)}(s,t) \, . $$ The marginal empirical distribution functions of each sample are $$ \hat{G}_{n_1}(s) = \frac{1}{n_1} \sum_{i=1}^{n_1} I_{[x_i,\infty)}(s) \quad \textrm{and} \quad \hat{H}_{n_2}(t) = \frac{1}{n_2} \sum_{i=1}^{n_2} I_{[y_i,\infty)}(t) \, . $$ The idea is to compare $\hat{F}_{n}(s,t)$ with the product $\hat{H}_{n_1}(s)\hat{G}_{n_2}(t)$ using some norm. For example, you could use $$ T(x,y) = \sup_{s,t} \bigg\vert \hat{F}_{n}(s,t) - \hat{G}_{n_1}(s)\hat{H}_{n_2}(t) \bigg\vert \, . $$ If we could know the distribution of $T$ under the hypothesis of independence, then we would have a way to compute a $p$-value for this problem. I don't know how this can be done.