3

I am trying to find if there exists a way to find the spatial distribution over a set of points where the points are weighted. If I have "n" points in the (x,y) space, I can fit a mixture of Gaussians to estimate a distribution. However, there might be situations where some points are more important than others so I have weights associated with my points. For example, the k-means algorithm can be extended to a weighted k-means problem. Is there a way to do so with a mixture of Gaussians?

SO has a post on this here but there are no accepted answers. I am having a similar problem and I thought this is a better forum.

Community
  • 1
statBeginner
  • 1,581
  • Before to write up a long answer - if you look closely into how EM works you will see that in each iteration $\vec{x}$ is used to compute the means per group. You can use the $\vec{x}\cdot\vec{w}_x$ there. Alternatively (or even additinally... need to think about it more) you need to include these terms into the computation of the log-likelihood. But given that... you can scale your samples according to the weights before applying EM and then run it. The result will be the same (because log-likelihood is used as objective function) – Drey Nov 25 '16 at 13:12

1 Answers1

-1

In case there is no "standard" weighted GMM in the literature (I'm suprised I haven't found anything), I would try the following:

Let:

  • $z_i$ be the cluster of point $i$
  • $w_i$ be the observed weight $[0, \inf)$
  • $\boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k$ the mean and covariance of the cluster $k$

and knowing that a non-weighted GMM considers that points in are generated according to:

\begin{align} \mathbf{x_i} \sim \mathcal{N}(\boldsymbol{\mu}_{z_i}, \boldsymbol{\Sigma}_{z_i})\\ \end{align}

you could modify this generative process a little bit and say that points with less weight have more freedom to choose their location. That is, you give them more variance. It might be:

\begin{align} \mathbf{x_i} \sim \mathcal{N}(\boldsymbol{\mu}_{z_i}, e^{w_i}\boldsymbol{\Sigma}_{z_i})\\ \end{align}

or

\begin{align} \mathbf{x_i} \sim \mathcal{N}(\boldsymbol{\mu}_{z_i}, (1+w_i)\boldsymbol{\Sigma}_{z_i})\\ \end{align}

alberto
  • 3,056
  • Thanks. I do not quite get the idea behind having freedom to choose locations. The points already have fixed locations, don't they? – statBeginner Mar 08 '15 at 18:42
  • Does Jakob's answer from the SO post make sense to you? Here is what we wrote : If $v_i$ is the weight of the i-th sample, the algorithm from the tutorial (see end of Section 6.2.) changes so that the $\gamma_{ij}$ is multiplied by that weighting factor. For the calculation of the new weights $w_j$, $n_j$ has to be divided by the sum of the weights $\sum_{i=1}^{n} v_i$ instead of just n. – statBeginner Mar 08 '15 at 18:43
  • If $\gamma$ is the probability of belonging to a cluster I don't see how multiplying by a weighting factor solves the problem. (Maybe, but it doesn't make sense to me, what is the interpretation?) – alberto Mar 08 '15 at 18:54
  • About the freedom. The trick is that your prior on $x_i$ is more flexible, less dogmatic, if $x_i$ has a small weight, so the algorithm wouldn't care as much about these points. – alberto Mar 08 '15 at 18:56
  • Thanks, I see what you are trying to say now. About multiplying the weights, i think what he is trying to say is that after you calculate the probabilities, you rescale them by multiplying them with the weights. Then to normalize, instead of dividing by $n$, you divide by the sum of weights. – statBeginner Mar 08 '15 at 19:25
  • But how would that change the probability of being assigned to one or other cluster? Maybe you could sketch the scheme in your answer so that we can discuss it better. – alberto Mar 08 '15 at 19:29