Music Spectrogram diffusion pipeline #1044

kashif · 2022-10-28T16:30:19Z

For issue #320 and #544

HuggingFaceDocBuilderDev · 2022-10-28T16:34:03Z

The documentation is not available anymore as the PR was closed or merged.

patrickvonplaten · 2022-10-31T18:56:06Z

@patil-suraj @anton-l maybe you could already take a look here if you find some time :-)

anton-l · 2022-11-02T13:57:37Z

There's no pipeline.__call__ yet, so I'll wait for @kashif's ping when it's ready 😄

kashif · 2022-11-02T14:02:33Z

yup getting there...

patrickvonplaten · 2022-11-04T17:40:54Z

This looks like a great start! @kashif could you add a code snippet showing how to run the model for inference? Similar to #658 (comment) maybe? :-)

Once we can reproduce some results locally, I think we'll have a much easier time getting this PR merged :-)

…on.py Co-authored-by: Patrick von Platen <[email protected]>

patrickvonplaten

Cool, I think we're very close to merging this one.

Only thing left to do is:

Update notes to adhere to the newest API (e.g. don't pass file name to pipeline, but tokens
Update tests and docs to use "google/...." model id instead of "kashif"`

Also could you maybe update the example here: https://huggingface.co/google/music-spectrogram-diffusion#example-usage

…on.py Co-authored-by: Patrick von Platen <[email protected]>

patrickvonplaten · 2023-03-21T13:06:25Z

@patrickvonplaten so the failing docs are because note_seq installs the latest protobuf which causes issues with tensorboard etc. so one solution was to also pin protobuf to 3.20.x which seems to make everyone happy?

Ahh I see - yes let's pin it :-)

tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py

…on.py

patrickvonplaten · 2023-03-21T14:26:19Z

Think we need a "make style" here and then we should be good to go :-)

kashif · 2023-03-21T16:15:57Z

@patrickvonplaten I could not reproduce the one failed mps test...

patrickvonplaten · 2023-03-23T12:25:03Z

Think the MPS failure is unrelated - let's merge once the tests now are more or less green :-)

* initial TokenEncoder and ContinuousEncoder * initial modules * added ContinuousContextTransformer * fix copy paste error * use numpy for get_sequence_length * initial terminal relative positional encodings * fix weights keys * fix assert * cross attend style: concat encodings * make style * concat once * fix formatting * Initial SpectrogramPipeline * fix input_tokens * make style * added mel output * ignore weights for config * move mel to numpy * import pipeline * fix class names and import * moved models to models folder * import ContinuousContextTransformer and SpectrogramDiffusionPipeline * initial spec diffusion converstion script * renamed config to t5config * added weight loading * use arguments instead of t5config * broadcast noise time to batch dim * fix call * added scale_to_features * fix weights * transpose laynorm weight * scale is a vector * scale the query outputs * added comment * undo scaling * undo depth_scaling * inital get_extended_attention_mask * attention_mask is none in self-attention * cleanup * manually invert attention * nn.linear need bias=False * added T5LayerFFCond * remove to fix conflict * make style and dummy * remove unsed variables * remove predict_epsilon * Move accelerate to a soft-dependency (huggingface#1134) * finish * finish * Update src/diffusers/modeling_utils.py * Update src/diffusers/pipeline_utils.py Co-authored-by: Anton Lozhkov <[email protected]> * more fixes * fix Co-authored-by: Anton Lozhkov <[email protected]> * fix order * added initial midi to note token data pipeline * added int to int tokenizer * remove duplicate * added logic for segments * add melgan to pipeline * move autoregressive gen into pipeline * added note_representation_processor_chain * fix dtypes * remove immutabledict req * initial doc * use np.where * require note_seq * fix typo * update dependency * added note-seq to test * added is_note_seq_available * fix import * added toc * added example usage * undo for now * moved docs * fix merge * fix imports * predict first segment * avoid un-needed copy to and from cpu * make style * Copyright * fix style * add test and fix inference steps * remove bogus files * reorder models * up * remove transformers dependency * make work with diffusers cross attention * clean more * remove @ * improve further * up * uP * Apply suggestions from code review * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py * loop over all tokens * make style * Added a section on the model * fix formatting * grammer * formatting * make fix-copies * Update src/diffusers/pipelines/__init__.py Co-authored-by: Patrick von Platen <[email protected]> * Update src/diffusers/pipelines/spectrogram_diffusion/pipeline_spectrogram_diffusion.py Co-authored-by: Patrick von Platen <[email protected]> * added callback ad optional ionnx * do not squeeze batch dim * clean up more * upload * convert jax to nnumpy * make style * fix warning * make fix-copies * fix warning * add initial fast tests * add initial pipeline_params * eval mode due to dropout * skip batch tests as pipeline runs on a single file * make style * fix relative path * fix doc tests * Update src/diffusers/models/t5_film_transformer.py Co-authored-by: Patrick von Platen <[email protected]> * Update src/diffusers/models/t5_film_transformer.py Co-authored-by: Patrick von Platen <[email protected]> * Update docs/source/en/api/pipelines/spectrogram_diffusion.mdx Co-authored-by: Patrick von Platen <[email protected]> * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py Co-authored-by: Patrick von Platen <[email protected]> * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py Co-authored-by: Patrick von Platen <[email protected]> * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py Co-authored-by: Patrick von Platen <[email protected]> * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py Co-authored-by: Patrick von Platen <[email protected]> * add MidiProcessor * format * fix org * Apply suggestions from code review * Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py * make style * pin protobuf to <4 * fix formatting * white space * tensorboard needs protobuf --------- Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Anton Lozhkov <[email protected]>

kashif added 3 commits October 26, 2022 18:22

initial TokenEncoder and ContinuousEncoder

f85d908

initial modules

e025410

added ContinuousContextTransformer

e88dc6f

kashif marked this pull request as draft October 28, 2022 16:30

Merge branch 'main' into spectrogram-diffusion

c9dd1dd

patrickvonplaten requested review from patrickvonplaten and anton-l October 31, 2022 18:55

fix copy paste error

59e2111

kashif added 13 commits November 2, 2022 21:57

use numpy for get_sequence_length

ab82923

initial terminal relative positional encodings

cdc6ec7

fix weights keys

c55fb5b

fix assert

af67374

cross attend style: concat encodings

ef43fe0

Merge branch 'main' into spectrogram-diffusion

33755df

make style

6de0cfb

Merge branch 'main' into spectrogram-diffusion

1068282

concat once

5546c12

fix formatting

8b32df3

Initial SpectrogramPipeline

c69a3b9

fix input_tokens

f7254db

make style

133d155

kashif added 4 commits November 7, 2022 16:41

added mel output

aa2323f

ignore weights for config

c154878

move mel to numpy

63f69b6

import pipeline

9808d06

kashif and others added 3 commits March 21, 2023 14:05

Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusi…

dd9f8ca

…on.py Co-authored-by: Patrick von Platen <[email protected]>

Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusi…

654c796

…on.py Co-authored-by: Patrick von Platen <[email protected]>

Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusi…

9a8a93d

…on.py Co-authored-by: Patrick von Platen <[email protected]>

patrickvonplaten approved these changes Mar 21, 2023

View reviewed changes

Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusi…

ebb8e9a

…on.py Co-authored-by: Patrick von Platen <[email protected]>

kashif added 3 commits March 21, 2023 14:28

add MidiProcessor

3a94476

format

7c43be8

fix org

6dcd3f7

patrickvonplaten reviewed Mar 21, 2023

View reviewed changes

tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Mar 21, 2023

View reviewed changes

tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Mar 21, 2023

View reviewed changes

tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py Outdated Show resolved Hide resolved

Apply suggestions from code review

17b7481

patrickvonplaten reviewed Mar 21, 2023

View reviewed changes

tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusion.py Outdated Show resolved Hide resolved

Update tests/pipelines/spectrogram_diffusion/test_spectrogram_diffusi…

458e7b7

…on.py

kashif added 6 commits March 21, 2023 15:31

make style

4f27f66

Merge branch 'main' into spectrogram-diffusion

dc8280e

pin protobuf to <4

76a28c1

fix formatting

7339d37

white space

f71b155

tensorboard needs protobuf

e5225a3

Merge branch 'main' into spectrogram-diffusion

8abbd57

patrickvonplaten merged commit 2ef9bdd into huggingface:main Mar 23, 2023

kashif deleted the spectrogram-diffusion branch March 23, 2023 13:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Music Spectrogram diffusion pipeline #1044

Music Spectrogram diffusion pipeline #1044

Uh oh!

kashif commented Oct 28, 2022 •

edited

Loading

Uh oh!

HuggingFaceDocBuilderDev commented Oct 28, 2022 •

edited

Loading

Uh oh!

patrickvonplaten commented Oct 31, 2022

Uh oh!

anton-l commented Nov 2, 2022

Uh oh!

kashif commented Nov 2, 2022

Uh oh!

patrickvonplaten commented Nov 4, 2022

Uh oh!

patrickvonplaten left a comment

Uh oh!

patrickvonplaten commented Mar 21, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrickvonplaten commented Mar 21, 2023

Uh oh!

kashif commented Mar 21, 2023

Uh oh!

patrickvonplaten commented Mar 23, 2023

Uh oh!

Uh oh!

Music Spectrogram diffusion pipeline #1044

Music Spectrogram diffusion pipeline #1044

Uh oh!

Conversation

kashif commented Oct 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Oct 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patrickvonplaten commented Oct 31, 2022

Uh oh!

anton-l commented Nov 2, 2022

Uh oh!

kashif commented Nov 2, 2022

Uh oh!

patrickvonplaten commented Nov 4, 2022

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Mar 21, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrickvonplaten commented Mar 21, 2023

Uh oh!

kashif commented Mar 21, 2023

Uh oh!

patrickvonplaten commented Mar 23, 2023

Uh oh!

Uh oh!

kashif commented Oct 28, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Oct 28, 2022 •

edited

Loading