Add Argmax DiffusionKit Snippet #869

ardaatahan · 2024-08-26T23:18:15Z

This PR:

Adds a new diffusionkit snippet in model-libraries-snippets.ts.
Adds diffusionkit library with necessary information in MODEL_LIBRARIES_UI_ELEMENTS in model-libraries.ts.

Wauplin

Looking great, thanks @ardaatahan! I've added some comments to improve the integration and then we should be good. Don't forget to tag models on the Hub as diffusionkit (we usually prefer to have a few examples before merging 😉)

Wauplin · 2024-08-27T09:01:07Z

packages/tasks/src/model-libraries-snippets.ts

@@ -905,4 +905,33 @@ whisperkit-cli transcribe --audio-path /path/to/audio.mp3
 # Or use your preferred model variant
 whisperkit-cli transcribe --model "large-v3" --model-prefix "distil" --audio-path /path/to/audio.mp3 --verbose`,
 ];
+
+export const diffusionkit = (): string[] => [


Can you move this section up in the file to preserve alphabetical order as much as possible? (just after diffusers for instance)

Wauplin · 2024-08-27T09:05:36Z

packages/tasks/src/model-libraries.ts

+		prettyLabel: "DiffusionKit",
+		repoName: "DiffusionKit",
+		repoUrl: "https://github.com/argmaxinc/DiffusionKit",
+		docsUrl: "https://github.com/argmaxinc/DiffusionKit?tab=readme-ov-file#-image-generation-with-python-mlx",


Suggested change

docsUrl: "https://github.com/argmaxinc/DiffusionKit?tab=readme-ov-file#-image-generation-with-python-mlx",

No need to provide a docsUrl when it's the same as repoUrl

Wauplin · 2024-08-27T09:06:49Z

packages/tasks/src/model-libraries.ts

@@ -174,6 +174,13 @@ export const MODEL_LIBRARIES_UI_ELEMENTS = {
 		filter: true,
 		/// diffusers has its own more complex "countDownloads" query
 	},
+	diffusionkit: {


Would be nice to have some models listed in https://huggingface.co/models?other=diffusionkit before merging this PR. You can do that by tagging models as diffusionkit in the model card metadata.

I opened a first PR to do that: https://huggingface.co/argmaxinc/mlx-FLUX.1-schnell/discussions/2

Merged 👍

I noticed that the SD3 models have the DiffusionKit tag: https://huggingface.co/models?other=DiffusionKit. The correct tag should be diffusionkit (lowercased). Make sure to update them to benefit from this PR.

(the correct key is the key defined here, so diffusionkit in this case. Below we are defining prettyLabel and repoName but that's just for aesthetic in the UI)

We generated these models with diffusionkit much before we had an MLX implementation, and in fact don't yet have a public CoreML inference implementation of DiffusionKit (in the works). These models are actually more directly usable by the huggingface swift-coreml-diffusers app, although they were generated with diffusionkit. So in this case we might want to just remove the lowercase diffusionkit tag? @pcuenca what do you think about a local-app deeplink for Diffusers.app? We had discussed adding arbitrary repos in the past.

Updated our tags and model cards: https://huggingface.co/models?other=diffusionkit

Wauplin · 2024-08-27T09:09:34Z

packages/tasks/src/model-libraries-snippets.ts

+	`# Install CLI with pip
+pip install diffusionkit-cli
+
+# View all available options
+diffusionkit-cli --help
+
+# Generate image using default FLUX.1-schnell and save it to out.png
+diffusionkit-cli --prompt "a beautiful sunset over the ocean"
+
+# To use Stable Diffusion 3 accept the terms before downloading the checkpoint: https://huggingface.co/stabilityai/stable-diffusion-3-medium
+# Once you accept the terms, sign in with your Hugging Face hub token with read access to contents of all public gated repos you can access:
+huggingface-cli login --token YOUR_HF_HUB_TOKEN
+


The "code snippet" part is generally a very short snippet to help users getting started. It should not contain installation or configuration steps and should be focused on "how to use this model with diffusionkit". Code snippets are shown on each model page and are customized to be ready to use.

Suggested change

`# Install CLI with pip

pip install diffusionkit-cli

# View all available options

diffusionkit-cli --help

# Generate image using default FLUX.1-schnell and save it to out.png

diffusionkit-cli --prompt "a beautiful sunset over the ocean"

# To use Stable Diffusion 3 accept the terms before downloading the checkpoint: https://huggingface.co/stabilityai/stable-diffusion-3-medium

# Once you accept the terms, sign in with your Hugging Face hub token with read access to contents of all public gated repos you can access:

huggingface-cli login --token YOUR_HF_HUB_TOKEN

`

Agree with @Wauplin. In addition, I'd recommend that different examples or use cases are returned as multiple strings, like in the audioseal example. This way snippets are separated visually and can be copied independently:

Even better yes!

Great example, was curious how to do this!

Wauplin · 2024-08-27T09:11:25Z

packages/tasks/src/model-libraries-snippets.ts

+# Use specific model and set custom output path
+diffusionkit-cli --prompt "a futuristic cityscape" --model-version stable-diffusion-3-medium --output-path /path/to/output.png


Suggested change

# Use specific model and set custom output path

diffusionkit-cli --prompt "a futuristic cityscape" --model-version stable-diffusion-3-medium --output-path /path/to/output.png

# Generate an image and save it to out.png

diffusionkit-cli --prompt "a futuristic cityscape" --model-version ${model.id}

Wauplin · 2024-08-27T09:14:18Z

packages/tasks/src/model-libraries-snippets.ts

+# Set seed for reproducibility, specify number of steps, and set custom output image dimensions
+diffusionkit-cli --prompt "detailed cinematic dof render of a \
+detailed MacBook Pro on a wooden desk in a dim room with items \
+around, messy dirty room. On the screen are the letters 'FLUX on \
+DiffusionKit' glowing softly. High detail hard surface render" \
+--height 768 \
+--width 1360 \
+--seed 1001 \
+--step 4 \
+--output ~/Desktop/flux_on_mac.png`,


Suggested change

# Set seed for reproducibility, specify number of steps, and set custom output image dimensions

diffusionkit-cli --prompt "detailed cinematic dof render of a \

detailed MacBook Pro on a wooden desk in a dim room with items \

around, messy dirty room. On the screen are the letters 'FLUX on \

DiffusionKit' glowing softly. High detail hard surface render" \

--height 768 \

--width 1360 \

--seed 1001 \

--step 4 \

--output ~/Desktop/flux_on_mac.png`,

# Specify dimensions, seed, number of steps and destination file

diffusionkit-cli \

--prompt "detailed cinematic dof render of a \

detailed MacBook Pro on a wooden desk in a dim room with items \

around, messy dirty room. On the screen are the letters 'FLUX on \

DiffusionKit' glowing softly. High detail hard surface render" \

--model-version ${model.id}

--height 768 \

--width 1360 \

--seed 1001 \

--step 4 \

--output ~/Desktop/flux_on_mac.png`,

Add --model-version ${model.id} + some nits

Vaibhavs10

Hey @ardaatahan - thanks for the PR. 🤗

Quick thought, for the libraries PR let's use the actual modeling code snippets i.e. https://github.com/argmaxinc/DiffusionKit?tab=readme-ov-file#code

This also makes it easy for people to quickly able to take the snippet and run (this is similar to how we show for transformers)

and then for the CLI we can do a separate LocalApps integration - local apps integration would be similar to how we currently show drawthings, and other Text to Image apps?

huggingface.js/packages/tasks/src/local-apps.ts

Line 219 in c367ae4

drawthings: {

I think this would give you the best distribution overall since you promote your library as a modeling/ conversion library plus as an app as well.

cc: @atiorh @enzostvs @pcuenca @Wauplin

Thoughts?

If you think this is a good idea then the next step would be to:

Replace the CLI code snippet here with Python inference snippets.
Open a new PR similar to how the drawthings integration I linked above is.

ZachNagengast · 2024-08-27T17:03:52Z

Agree with everything here, will adjust based on review.

@Vaibhavs10 thanks for your comment, also curious about thoughts from the rest of the team. I think its a great suggestion. One caveat is that the mlx implementation of diffusionkit is not really a fully fledge app, but more of a cli utility into the python library (similar to howwhisperkit-cli provides an interface into the swift library). On the other hand, it will work with the source repos of some very specific text-to-image models that we support so far (flux and sd3) since we do the model dict adjustments inside the library, so will investigate how to filter that down similar to drawthings.

Vaibhavs10

app, but more of a cli utility into the python library (similar to howwhisperkit-cli provides an interface into the swift library). On the other hand, it will work with the source repos of some very specific text-to-image models that we support so far (flux and sd3) since we do the model dict adjustments inside the library, so will investigate how to filter that down similar to drawthings.

Ah makes sense - the current structure makes sense to me with actual model inference snippets over CLI. We can discuss CLI LocalApps integration a bit later once DiffusionKit matures a bit more potentially.

Reviewing the PR real quick.

Vaibhavs10

LGTM! Much cleaner! 🤗

Note: There's a little conflict, make sure to rebase!

pcuenca · 2024-08-28T12:10:08Z

packages/tasks/src/model-libraries-snippets.ts

+pipeline = DiffusionPipeline(
+	shift=3.0,
+	use_t5=False,
+	model_version="argmaxinc/mlx-stable-diffusion-3-medium",


So is it not possible to do it model-specific at this time, as @Wauplin mentioned?

Agree that it would be best to showcase only the relevant code snippet here. How does the CLI determines which pipeline to instantiate? I would reuse the same logic here. Otherwise, it's also possible to tag the models on the Hub as "flux" or stable-diffusion (using the tags list in the model card metadata) and use this information to generate the snippets.

Thank you for your feedback! Both of your proposals are excellent suggestions. I will be implementing the latter approach, using tags in the model card metadata to determine which snippets to generate.

pcuenca · 2024-08-28T12:11:17Z

packages/tasks/src/model-libraries-snippets.ts

+  latent_size=(HEIGHT // 8, WIDTH // 8),
+)`;
+
+	return [sd3Snippet, fluxSnippet, generateSnippet];


Since we have multiple snippets, it may be useful to include a comment line at the beginning of each one to explain what they are about.

Agreed and added, thanks!

Wauplin

Hey! Sorry being late to the party. Let's add it only as a library for now. Note that since we are displaying multiple snippets, it is still possible to have "Python" snippets followed by "CLI" snippets.

Wauplin · 2024-08-28T12:17:57Z

packages/tasks/src/model-libraries-snippets.ts

+pipeline = DiffusionPipeline(
+	shift=3.0,
+	use_t5=False,
+	model_version="argmaxinc/mlx-stable-diffusion-3-medium",


Agree that it would be best to showcase only the relevant code snippet here. How does the CLI determines which pipeline to instantiate? I would reuse the same logic here. Otherwise, it's also possible to tag the models on the Hub as "flux" or stable-diffusion (using the tags list in the model card metadata) and use this information to generate the snippets.

…LEMENTS

ardaatahan · 2024-08-28T18:22:40Z

app, but more of a cli utility into the python library (similar to howwhisperkit-cli provides an interface into the swift library). On the other hand, it will work with the source repos of some very specific text-to-image models that we support so far (flux and sd3) since we do the model dict adjustments inside the library, so will investigate how to filter that down similar to drawthings.

Ah makes sense - the current structure makes sense to me with actual model inference snippets over CLI. We can discuss CLI LocalApps integration a bit later once DiffusionKit matures a bit more potentially.

Reviewing the PR real quick.

Hey @Vaibhavs10, thank you for your feedback! After your original comment about opening a new PR similar to how the drawthings integration is, I made similar changes to the local-apps file for DiffusionKit and I'm curious about your thoughts:

+ const snippetDiffusionKit = (model: ModelData): LocalAppSnippet[] => {
+	const command = (binary: string) =>
+		[
+			"# Load and run the model:",
+			`${binary} \\`,
+			`  --model-version ${model.id}" \\`,
+			'  --prompt "a futuristic cityscape" \\',
+			`  --height 768 \\`,
+			`  --width 1360 \\`,
+			`  --seed 1001 \\`,
+			`  --step 4 \\`,
+			`  --output ~/Desktop/out.png`,
+		].join("\n");
+	return [
+		{
+			title: "Install from pip",
+			setup: "pip install diffusionkit",
+			content: command("diffusionkit-cli"),
+		},
+		{
+			title: "Build from source code",
+			setup: [
+				// prettier-ignore
+				"git clone https://github.com/argmaxinc/DiffusionKit.git",
+				"cd DiffusionKit",
+				"pip install -e .",
+			].join("\n"),
+			content: command("diffusionkit-cli"),
+		},
+	];
+ };

Inside LOCAL_APPS:

+ diffusionkit: {
+		prettyLabel: "DiffusionKit",
+		docsUrl: "https://github.com/argmaxinc/DiffusionKit",
+		mainTask: "text-to-image",
+		macOSOnly: true,
+		displayOnModelPage: (model) => model.library_name === "diffusionkit" && model.pipeline_tag === "text-to-+image",
+		snippet: snippetDiffusionKit,
+ },

Given that these local app changes are already implemented, would you recommend creating a new PR to handle them?

Wauplin

Thanks for the changes! :)

Wauplin · 2024-08-29T09:03:25Z

packages/tasks/src/model-libraries-snippets.ts

+NUM_STEPS = 4  #  4 for FLUX.1-schnell, 50 for SD3
+CFG_WEIGHT = 0. # for FLUX.1-schnell, 5. for SD3


Suggested change

NUM_STEPS = 4 # 4 for FLUX.1-schnell, 50 for SD3

CFG_WEIGHT = 0. # for FLUX.1-schnell, 5. for SD3

NUM_STEPS = ${model.tags.includes("flux") ? 4 : 50}

CFG_WEIGHT = ${model.tags.includes("flux") ? 0. : 5}

Let's reuse the flux tag to generate the pipeline config as well :)

(disclaimer: not tested)

Thanks for the suggestion! I just tested and made the changes.

packages/tasks/src/model-libraries-snippets.ts

Wauplin · 2024-08-29T09:06:22Z

packages/tasks/src/model-libraries.ts

@@ -174,6 +174,13 @@ export const MODEL_LIBRARIES_UI_ELEMENTS = {
 		filter: true,
 		/// diffusers has its own more complex "countDownloads" query
 	},
+	diffusionkit: {


I noticed that the SD3 models have the DiffusionKit tag: https://huggingface.co/models?other=DiffusionKit. The correct tag should be diffusionkit (lowercased). Make sure to update them to benefit from this PR.

(the correct key is the key defined here, so diffusionkit in this case. Below we are defining prettyLabel and repoName but that's just for aesthetic in the UI)

…ccording to model tags

ardaatahan · 2024-08-30T07:09:49Z

@Vaibhavs10 @pcuenca @Wauplin Let me know if there's any other changes needed here or if it is good to go. Open to any suggestions you'd think would help the users!

Wauplin

Everything looks good to me!
@Vaibhavs10 and/or @pcuenca please have a last check as well and we should be good to merge :)

pcuenca

Looks good to me, thanks a lot for iterating here 🙌 Let's wait for @Wauplin and @Vaibhavs10 thoughts!

Edit: had the review tab open while @Wauplin was submitting his review.

pcuenca · 2024-08-30T08:02:40Z

packages/tasks/src/model-libraries-snippets.ts

+
+	const pipelineSnippet = model.tags.includes("flux") ? fluxSnippet : sd3Snippet;
+
+	return [pipelineSnippet, generateSnippet];


We usually show a single snippet per task (in this case it would be pipeline preparation followed by generation), but I'm not opposed to using two to differentiate instantiation from generation.

packages/tasks/src/model-libraries-snippets.ts

pcuenca · 2024-08-30T08:43:17Z

Linter fixed, but two tests failing. Is it ok to merge @Wauplin @SBrandeis ?

Wauplin · 2024-08-30T08:52:00Z

Thanks for the linter :) I would say yes since the failing tests are unrelated to this PR (already happening on main and other PRs)

ardaatahan requested review from osanseviero, SBrandeis, gary149, Wauplin, julien-c and pcuenca as code owners August 26, 2024 23:18

Wauplin reviewed Aug 27, 2024

View reviewed changes

Vaibhavs10 reviewed Aug 27, 2024

View reviewed changes

Vaibhavs10 reviewed Aug 28, 2024

View reviewed changes

Vaibhavs10 approved these changes Aug 28, 2024

View reviewed changes

pcuenca reviewed Aug 28, 2024

View reviewed changes

Wauplin reviewed Aug 28, 2024

View reviewed changes

ardaatahan added 5 commits August 28, 2024 09:26

add diffusionkit snippet and add diffusionkit to MODEL_LIBRARIES_UI_E…

7c47f27

…LEMENTS

remove last 3 prompts and add a new one

44a5f8b

move hf login instructions down

57c93ca

fix snippet content and move it up to preserve alphabetical order

031b28d

show relevant pipeline according to tag

6055b69

ardaatahan force-pushed the add-diffusionkit-snippet branch from 1c0d5a9 to 6055b69 Compare August 28, 2024 18:03

Wauplin reviewed Aug 29, 2024

View reviewed changes

ardaatahan mentioned this pull request Aug 29, 2024

Add diffusionkit to local-apps #875

Open

ardaatahan added 2 commits August 29, 2024 14:26

Merge branch 'huggingface:main' into add-diffusionkit-snippet

9d55cdd

dynamically change num_steps and cfg_weight in diffusionkit snippet a…

49b810c

…ccording to model tags

Wauplin approved these changes Aug 30, 2024

View reviewed changes

pcuenca approved these changes Aug 30, 2024

View reviewed changes

pcuenca reviewed Aug 30, 2024

View reviewed changes

packages/tasks/src/model-libraries-snippets.ts Outdated Show resolved Hide resolved

Linter

6b983bb

Wauplin merged commit 94cb7fe into huggingface:main Aug 30, 2024
2 of 4 checks passed

		# Use specific model and set custom output path
		diffusionkit-cli --prompt "a futuristic cityscape" --model-version stable-diffusion-3-medium --output-path /path/to/output.png

		NUM_STEPS = 4 # 4 for FLUX.1-schnell, 50 for SD3
		CFG_WEIGHT = 0. # for FLUX.1-schnell, 5. for SD3


		const pipelineSnippet = model.tags.includes("flux") ? fluxSnippet : sd3Snippet;

		return [pipelineSnippet, generateSnippet];

Add Argmax DiffusionKit Snippet #869

Add Argmax DiffusionKit Snippet #869

Uh oh!

Conversation

ardaatahan commented Aug 26, 2024

Uh oh!

Wauplin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Vaibhavs10 left a comment

Choose a reason for hiding this comment

Uh oh!

ZachNagengast commented Aug 27, 2024

Uh oh!

Vaibhavs10 left a comment

Choose a reason for hiding this comment

Uh oh!

Vaibhavs10 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Wauplin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ardaatahan commented Aug 28, 2024

Uh oh!

Wauplin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ardaatahan commented Aug 30, 2024

Uh oh!

Wauplin left a comment

Choose a reason for hiding this comment

Uh oh!

Wauplin left a comment •

edited

Loading

Vaibhavs10 left a comment •

edited

Loading

pcuenca left a comment •

edited

Loading