ort integration #916

prathikr · 2022-10-20T00:25:07Z

Integrated ORTModule into examples/unconditional_image_generation/train_unconditional.py

Note: We required a change to unet_2d.py to return a raw tensor as output of the model instead of a dataclass. I verified that the changes to remove this dataclass do not affect baseline execution.

test command:

accelerate launch train_unconditional.py \
  --dataset_name="huggan/flowers-102-categories" \
  --resolution=64 \
  --output_dir="ddpm-ema-flowers-64" \
  --train_batch_size=16 \
  --num_epochs=1 \
  --gradient_accumulation_steps=1 \
  --learning_rate=1e-4 \
  --lr_warmup_steps=500 \
  --mixed_precision=no \
  --ort

HuggingFaceDocBuilderDev · 2022-10-20T22:06:52Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

patrickvonplaten · 2022-10-21T15:49:32Z

Leaving it up to @anton-l , maybe you could take a look whenever you find some time :-)

Signed-off-by: Ryan Russell <[email protected]>

* documenting `attention_flax.py` file * documenting `embeddings_flax.py` * documenting `unet_blocks_flax.py` * Add new objs to doc page * document `vae_flax.py` * Apply suggestions from code review * modify `unet_2d_condition_flax.py` * make style * Apply suggestions from code review * make style * Apply suggestions from code review * fix indent * fix typo * fix indent unet * Update src/diffusers/models/vae_flax.py * Apply suggestions from code review Co-authored-by: Pedro Cuenca <[email protected]> Co-authored-by: Mishig Davaadorj <[email protected]> Co-authored-by: Pedro Cuenca <[email protected]>

the result of running the pipeline is stored in StableDiffusionPipelineOutput.images

* refactor: pipelines readability improvements Signed-off-by: Ryan Russell <[email protected]> * docs: remove todo comment from flax pipeline Signed-off-by: Ryan Russell <[email protected]> Signed-off-by: Ryan Russell <[email protected]>

Fix "ort is not defined" issue.

* docs: `src/diffusers` readability improvements Signed-off-by: Ryan Russell <[email protected]> * docs: `make style` lint Signed-off-by: Ryan Russell <[email protected]> Signed-off-by: Ryan Russell <[email protected]>

…ce#627) fix formula for noise levels in karras scheduler and tests

…ce#447) (huggingface#472) * Return encoded texts by DiffusionPipelines * Updated README to show hot to use enoded_text_input * Reverted examples in README.md * Reverted all * Warning for long prompts * Fix bugs * Formatted

the link points to an old location of the train_unconditional.py file

* Remove deprecated `torch_device` kwarg. * Remove unused imports.

Signed-off-by: Ryan Russell <[email protected]> Signed-off-by: Ryan Russell <[email protected]>

* WIP: flax FlaxDiffusionPipeline & FlaxStableDiffusionPipeline * todo comment * Fix imports * Fix imports * add dummies * Fix empty init * make pipeline work * up * Allow dtype to be overridden on model load. This may be a temporary solution until huggingface#567 is addressed. * Convert params to bfloat16 or fp16 after loading. This deals with the weights, not the model. * Use Flax schedulers (typing, docstring) * PNDM: replace control flow with jax functions. Otherwise jitting/parallelization don't work properly as they don't know how to deal with traced objects. I temporarily removed `step_prk`. * Pass latents shape to scheduler set_timesteps() PNDMScheduler uses it to reserve space, other schedulers will just ignore it. * Wrap model imports inside availability checks. * Optionally return state in from_config. Useful for Flax schedulers. * Do not convert model weights to dtype. * Re-enable PRK steps with functional implementation. Values returned still not verified for correctness. * Remove left over has_state var. * make style * Apply suggestion list -> tuple Co-authored-by: Suraj Patil <[email protected]> * Apply suggestion list -> tuple Co-authored-by: Suraj Patil <[email protected]> * Remove unused comments. * Use zeros instead of empty. Co-authored-by: Mishig Davaadorj <[email protected]> Co-authored-by: Mishig Davaadorj <[email protected]> Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Suraj Patil <[email protected]>

* fix accelerate for testing * fix copies * uP

…ce#874) Update README.md

* [CI] Add Apple M1 tests * setup-python * python build * conda install * remove branch * only 3.8 is built for osx-arm * try fetching prebuilt tokenizers * use user cache * update shells * Reports and cleanup * -> MPS * Disable parallel tests * Better naming * investigate worker crash * return xdist * restart * num_workers=2 * still crashing? * faulthandler for segfaults * faulthandler for segfaults * remove restarts, stop on segfault * torch version * change installation order * Use pre-RC version of PyTorch. To be updated when it is released. * Skip crashing test on MPS, add new one that works. * Skip cuda tests in mps device. * Actually use generator in test. I think this was a typo. * make style Co-authored-by: Pedro Cuenca <[email protected]>

Fix autoencoder test.

* fix accelerate for testing * fix copies * uP

…ce#874) Update README.md

* [CI] Add Apple M1 tests * setup-python * python build * conda install * remove branch * only 3.8 is built for osx-arm * try fetching prebuilt tokenizers * use user cache * update shells * Reports and cleanup * -> MPS * Disable parallel tests * Better naming * investigate worker crash * return xdist * restart * num_workers=2 * still crashing? * faulthandler for segfaults * faulthandler for segfaults * remove restarts, stop on segfault * torch version * change installation order * Use pre-RC version of PyTorch. To be updated when it is released. * Skip crashing test on MPS, add new one that works. * Skip cuda tests in mps device. * Actually use generator in test. I think this was a typo. * make style Co-authored-by: Pedro Cuenca <[email protected]>

Fix autoencoder test.

…ikr/diffusers into prathikrao/ort-integration

prathikr · 2022-10-26T23:31:35Z

Hi @anton-l, I've tried rebasing off main a couple times to get the CI tests running but for some reason it says I need to resolve conflicts in tests/test_models_unet.py even though I believe it is merge-able. Could you please let me know what I am doing wrong? Thank you.

patrickvonplaten · 2022-10-29T07:25:40Z

Hey @prathikr,

Thanks a lot for the PR. We are trying to keep our examples as easy to understand as possible and I think we don't want to mix ORT and PyTorch in the same training example to keep the PyTorch simple and independent from ORT.

Could we maybe instead add a ORT compatible as its own standalone script?

Cc @anton-l

prathikr · 2022-10-31T19:03:17Z

Hi @patrickvonplaten ,

Absolutely, I just made the changes but the CI still seems to have issues with tests/test_models_unet.py. I'd like to highlight my changes to the unet_2d.py class. Is this safe to do? I've done my best to verify that removing the dataclass for the output of forward() doesn't affect downstream code if .sample is removed, but it might be best to have someone more familiar with the codebase confirm.

Thanks again,
Prathik

prathikr · 2022-11-01T00:22:56Z

@JingyaHuang could you please take a look at this PR?

patrickvonplaten

This looks good to me - @anton-l what do you think?

patrickvonplaten · 2022-11-02T16:06:16Z

@prathikr however it seems like the commit history of this PR is a bit messed up - could you maybe open a new PR with the intended changes 😅 - thanks!

anton-l · 2022-11-02T16:20:41Z

I'll need a second opinion from @JingyaHuang (we've discussed ORT training recently), don't have enough exp with training myself yet :)

prathikr · 2022-11-02T18:44:58Z

@patrickvonplaten I've created a new PR and tagged you three #1110

Prathik Rao added 10 commits September 22, 2022 13:45

wrapped model in ORTModule

ae151ff

bug fix

0624ebd

bug fixes

7a68077

formatting

62c0493

bug fix

d237f0f

remove commented blocks

cc09d21

bug fix

3216e7e

bug fix

9ca7106

formatting

eafdeec

add onnxruntime.training

6bd9120

patrickvonplaten requested a review from anton-l October 20, 2022 17:07

prathikr marked this pull request as ready for review October 20, 2022 22:02

Merge branch 'main' into prathikrao/ort-integration

051aa83

ryanrussell and others added 15 commits October 26, 2022 15:09

docs: .md readability fixups (huggingface#619)

abe1637

Signed-off-by: Ryan Russell <[email protected]>

fix docs: change sample to images (huggingface#613)

78ae5a2

the result of running the pipeline is stored in StableDiffusionPipelineOutput.images

Allow passing session_options for ORT backend (huggingface#620)

f80643e

Fix breaking error: "ort is not defined" (huggingface#626)

dfcc372

Fix "ort is not defined" issue.

docs: src/diffusers readability improvements (huggingface#629)

835279e

* docs: `src/diffusers` readability improvements Signed-off-by: Ryan Russell <[email protected]> * docs: `make style` lint Signed-off-by: Ryan Russell <[email protected]> Signed-off-by: Ryan Russell <[email protected]>

Fix formula for noise levels in Karras scheduler and tests (huggingfa…

9bd5f62

…ce#627) fix formula for noise levels in karras scheduler and tests

[CI] Fix onnxruntime installation order (huggingface#633)

c4619c0

Fix docs link to train_unconditional.py (huggingface#642)

dea0c7b

the link points to an old location of the train_unconditional.py file

Remove deprecated torch_device kwarg (huggingface#623)

c9a729d

* Remove deprecated `torch_device` kwarg. * Remove unused imports.

refactor: custom_init_isort readability fixups (huggingface#631)

f8c99ef

Signed-off-by: Ryan Russell <[email protected]> Signed-off-by: Ryan Russell <[email protected]>

Remove inappropriate docstrings in LMS docstrings. (huggingface#634)

b43ed93

prathikr marked this pull request as ready for review October 26, 2022 23:22

patrickvonplaten and others added 16 commits October 26, 2022 16:23

[Tests] Add accelerate to testing (huggingface#729)

01392ee

* fix accelerate for testing * fix copies * uP

Created using Colaboratory

05d2f08

remove bogus folder no.2

7a6f2df

Add generic inference example to community pipeline readme (huggingfa…

910e821

…ce#874) Update README.md

Fix autoencoder test (huggingface#886)

36c671e

Fix autoencoder test.

rebase

0e087d5

[Tests] Add accelerate to testing (huggingface#729)

a3d8758

* fix accelerate for testing * fix copies * uP

Created using Colaboratory

3782e87

remove bogus folder no.2

82781f4

Add generic inference example to community pipeline readme (huggingfa…

88c39c5

…ce#874) Update README.md

Fix autoencoder test (huggingface#886)

d50f071

Fix autoencoder test.

rebase

c50019f

remove random code add

e8e2917

Merge branch 'prathikrao/ort-integration' of https://github.com/prath…

17ab2d3

…ikr/diffusers into prathikrao/ort-integration

seperate script for ort

d6df970

Prathik Rao added 2 commits October 31, 2022 15:17

formatting

67243b4

removed random eval statement

eafdd59

anton-l requested a review from JingyaHuang November 1, 2022 01:11

patrickvonplaten reviewed Nov 2, 2022

View reviewed changes

prathikr closed this Nov 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ort integration #916

ort integration #916

Uh oh!

prathikr commented Oct 20, 2022 •

edited

Loading

Uh oh!

HuggingFaceDocBuilderDev commented Oct 20, 2022

Uh oh!

patrickvonplaten commented Oct 21, 2022

Uh oh!

prathikr commented Oct 26, 2022

Uh oh!

patrickvonplaten commented Oct 29, 2022

Uh oh!

prathikr commented Oct 31, 2022 •

edited

Loading

Uh oh!

prathikr commented Nov 1, 2022

Uh oh!

patrickvonplaten left a comment

Uh oh!

patrickvonplaten commented Nov 2, 2022

Uh oh!

anton-l commented Nov 2, 2022

Uh oh!

prathikr commented Nov 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

ort integration #916

ort integration #916

Uh oh!

Conversation

prathikr commented Oct 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Oct 20, 2022

Uh oh!

patrickvonplaten commented Oct 21, 2022

Uh oh!

prathikr commented Oct 26, 2022

Uh oh!

patrickvonplaten commented Oct 29, 2022

Uh oh!

prathikr commented Oct 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

prathikr commented Nov 1, 2022

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Nov 2, 2022

Uh oh!

anton-l commented Nov 2, 2022

Uh oh!

prathikr commented Nov 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

prathikr commented Oct 20, 2022 •

edited

Loading

prathikr commented Oct 31, 2022 •

edited

Loading