4

In deep learning, especially generative models, sometimes we need to add some random noise to the input of model. To make the sampling of random noise learnable (or differentiable), we need to reparameterize the sampling, just like diffusion models do.

For example, given x ~ N (μ, σ2), if we sampled x directly, the mean and variance would not be differentiable. In practice we need to reparameterize it with x = μ + σ*ε, where ε ~ N (0, 1). So we sample ε instead of x, making μ and σ differentiable and learnable.

But if given x ~ Poisson (λ), how to reparameterize it to make λ learnable?

Lorin60
  • 91

1 Answers1

3

You can re-parameterize $x$ by using the quantile function of the Poisson distribution.

$$x = Q(\epsilon;\lambda) \qquad \text{where $\epsilon \sim U(0,1)$}$$

If you also want the function to be differentiable then you might want to consider a gamma distribution (with a continuous support) instead of a Poisson distribution (with a discrete support).

Compare below the Poisson distribution and the gamma distribution (with fixed scale=1) for equal values of the mean. The variance as function of the mean is in both cases $\text{Var}(X) = \text{Mean}(X)$.

comparison of gamma and Poisson

Sextus Empiricus
  • 77,915
  • I may not use Gamma distribution because its probability density function differs from Poisson. If there are no precise methods, perhaps approximating the Poisson distribution by a Gaussian is better when lambda is big. – Lorin60 Jun 14 '23 at 13:37
  • 2
    @Lorin60 the Poisson has $\text{Var}(X) = \text{Mean}(X)$ whereas the Gamma has $\text{Var}(X) \propto \text{Mean}(X)^2$ if the shape parameter is fixed. But if you keep the scale of the gamma distribution fixed and equal to one, then the Gamma distribution has the same behaviour $\text{Var}(X) = \text{Mean}(X)$. (In a glm framework they would be the same) – Sextus Empiricus Jun 14 '23 at 16:32
  • An alternative is to use the conjugate prior for the shape parameter, which has the same form as the Poisson distribution, except that the discrete factorial is replaced by a gamma function. But that distribution may not be so easy to work with. – Sextus Empiricus Jun 14 '23 at 16:32
  • Thanks. I see. Gamma is a good approximation of Poisson when the scale of Gamma equals to 1. – Lorin60 Jun 15 '23 at 06:26
  • @Lorin60 and with different scales you can consider it as an over/under-dispersed Poisson distribution. – Sextus Empiricus Jun 15 '23 at 06:44
  • btw. I wonder why you would actually need a Poisson distribution as choice for the additional random noise. – Sextus Empiricus Jun 15 '23 at 06:47
  • I‘m simulating the process of a detector receiving photons, which is a Poisson process. So it would generate noticeable Poisson noise in resulting images when the number of photons is relatively small. – Lorin60 Jun 15 '23 at 06:58
  • @Lorin60 and you are coupling this to some deep learning model? In that case you may not have to make the model output directly the number of photons and instead you might have as model output the expected rate $\lambda$. Like in a classification model where we may have the model output probabilities instead of a single class. – Sextus Empiricus Jun 15 '23 at 07:43
  • That's it! I'm trying to do that. You have a deep understanding of it. :-) – Lorin60 Jun 15 '23 at 08:16
  • @Lorin60 here is a link to a post that I had in my mind when speaking about classification and using the probabities instead of the final classification https://stats.stackexchange.com/a/312787/164061 – Sextus Empiricus Jun 15 '23 at 08:34
  • It's quite right. I'm a biomedical engineer practitioner, so I can relate to the question and answers. – Lorin60 Jun 15 '23 at 09:02