Using Generative Adversarial Networks for joint distribution estimation

Question

I am trying to use GAN model to generate N-dimensional samples with joint probability distribution that looks like some training data. I am having trouble getting the probability distribution of the generated data to match the training data.

As a training example, I created a 2-dimensional training dataset from bimodal Gaussian distribution as shown here:

I am using dense layers for both generator and discriminator networks. Like this

GenerativeAdversarialNetwork(
  (generator): Sequential(
    (0): Dropout(p=0.2, inplace=False)
    (1): Linear(in_features=100, out_features=100, bias=True)
    (2): Tanh()
    (3): Linear(in_features=100, out_features=100, bias=True)
    (4): Tanh()
    (5): Linear(in_features=100, out_features=100, bias=True)
    (6): Tanh()
    (7): Linear(in_features=100, out_features=100, bias=True)
    (8): Tanh()
    (9): Linear(in_features=100, out_features=2, bias=True)
    (10): Tanh()
    (11): Linear(in_features=2, out_features=2, bias=True)
  )
  (discriminator): Sequential(
    (0): Linear(in_features=2, out_features=32, bias=True)
    (1): LeakyReLU(negative_slope=0.01)
    (2): Linear(in_features=32, out_features=32, bias=True)
    (3): LeakyReLU(negative_slope=0.01)
    (4): Linear(in_features=32, out_features=32, bias=True)
    (5): LeakyReLU(negative_slope=0.01)
    (6): Linear(in_features=32, out_features=16, bias=True)
    (7): LeakyReLU(negative_slope=0.01)
    (8): Dropout(p=0.2, inplace=False)
    (9): Linear(in_features=16, out_features=1, bias=True)
    (10): Sigmoid()
  )
)

The closest that I could get was some bimodal distribution that is densely located at the two centers of the bimodal Gaussian distributions from the real data.

I have tried using the following recommendations from here and other places without any success.

• Using Adam optimizer
• Using dropout layers
• Playing with bigger networks
• Playing with learning rates
• Playing with different number of iterations or thresholds for training generator and discriminator networks.

My question is twofold:

• Is there anything I am missing?
• I have mainly seen people using GANs for image generation and claiming that the network learns the PDFs. But all the claims about the learned PDFs are qualitative, such as claiming that the generated images look good enough. Is there any demonstration of the actually matching PDFs. Such as 1D or 2D datapoints that can be easily visualized. It is easy for me to imagine that the generated samples are from some areas in the total distribution (like the orange samples that I am generating above) but it does not capture the total distribution at all.

Sorry about the long question.

Answer 1

GANs are generally known to be hard to train and to collapse into producing only one "kind" of samples. They work well on images, because the discriminator forces the generator to produce good looking images. But usually (in my experience) you will have a hard time to get a GAN to produce the correct distribution. I think normalizing flows or diffusion models are way more likely to solve that problem.

Answer 2

I have mainly seen people using GANs for image generation and claiming that the network learns the PDFs. But all the claims about the learned PDFs are qualitative, such as claiming that the generated images look good enough. Is there any demonstration of the actually matching PDFs. Such as 1D or 2D datapoints that can be easily visualized. It is easy for me to imagine that the generated samples are from some areas in the total distribution (like the orange samples that I am generating above) but it does not capture the total distribution at all.

One of the issues with being strictly quantitative in the comparison of the PDFs is that they aren't the same, and large sample sizes will catch small differences when they are brought to hypothesis tests (as they should, I argue). Thus, how close the distributions have to be for the GAN to be useful comes down to a judgment call. Sure, that can be quantified with something like KL divergence, but KL divergence lacks the intuition that you get from looking at synthetic images that a human cannot distinguish from real images, particularly when the litmus test probably is if a human can tell the difference.