[WIP] Add DDGAN Pipeline and Scheduler

[dg845]

What does this PR do?

This PR adds a pipeline and scheduler for the denoising diffusion GAN (DDGAN) model (paper, code). The GAN discriminator is not implemented in this PR.

Partially addresses #5905.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten
@sayakpaul
@JunbongJang

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[dg845]

I'm not sure if adding a new pipeline and scheduler to support DDGAN is strictly necessary - I believe DDGANPipeline + DDGANScheduler is very similar to DDPMPipeline + DDPMScheduler, with the following possible issues:

• The GAN generator denoising model might not be expressible as a UNet2DModel; the original DDGAN paper uses a variant of the NCSN++ architecture (see Appendix C.1).
• The original DDGAN implementation uses a different definition of beta_start and beta_end: I believe it gives these quantities in terms of what diffusers calls 1.0 - self.alphas_cumprod (and what the original DDPM paper would call $1 - \bar\alpha_t$, which is the variance of $q(x_t \mid x_0)$ rather than of $q(x_t \mid x_{t - 1})$). Accordingly, it implements different variance schedules (geometric and variance preserving). I haven't figured out yet if these variance schedules are equivalent to beta schedules already implemented in DDPMScheduler.

I think if these issues could be resolved (possibly with some additions to DDPMScheduler) we could use existing pipelines and schedulers for DDGAN checkpoints.

[patil-suraj]

Amazing @dg845 ! I also have a script ready but didn't manage to get good results yet. Have you already tried some training runs ?

[dg845]

@patil-suraj I haven't looked into implementing DDGAN training yet, this PR so far only implements a pipeline and scheduler for DDGAN inference.

Happy to look at/help with a DDGAN training script if that sounds good to you :).

[patil-suraj]

cool I will share the script with you (or push in this PR) later today.

[patil-suraj]

Oh actually I meant I have the script for https://arxiv.org/abs/2311.09257 which is formulated slightly differently than DDGAN

[dg845]

I am also planning to open a PR for a UFOGen scheduler (and pipeline if necessary) in light of #5905 but decided to work on a DDGAN pipeline and scheduler first since there is official code and checkpoints available (unlike for UFOGen, unless I'm missing something).

Would still be happy to take a look at the UFOGen training script :).

[patil-suraj]

Yeah will share it either way. The checkpoints are not available but looking into reproducing UFOGen.

[XiaXin-Aloys]

patil-suraj
Dear @patil-suraj @dg845,

I saw your mention of having a script for the UFOGen model. I'm studying text-to-image generation and your script could be incredibly helpful. I assure you it'll be used responsibly for my research. Could you consider sharing it? I appreciate your help.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.