clip-guided stable diffusion correctness

[keturn]

Describe the bug

the clip-guided pipeline uses these text-embeddings:

diffusers/examples/community/clip_guided_stable_diffusion.py

Lines 285 to 288 in 2345481

	# perform clip guidance
	if clip_guidance_scale > 0:
	text_embeddings_for_guidance = (
	text_embeddings.chunk(2)[0] if do_classifier_free_guidance else text_embeddings

defined earlier as:

diffusers/examples/community/clip_guided_stable_diffusion.py

Lines 234 to 237 in 2345481

	# For classifier free guidance, we need to do two forward passes.
	# Here we concatenate the unconditional and text embeddings into a single batch
	# to avoid doing two forward passes
	text_embeddings = torch.cat([uncond_embeddings, text_embeddings])

which I read as using the unconditioned (i.e. null prompt) embeddings.

Is that the way it's supposed to work? That doesn't feel like how it's supposed to work. Like, if the normal classifier-free guidance function is turned off, it would be using the embeddings from the text prompt, not the nulls.

But this pipeline was added without any tests or samples or other reference material, so I really don't know.

Reproduction

No response

Logs

No response

System Info

👀

[patrickvonplaten]

cc @patil-suraj might be good to take a look soon since we just released it :-)

[keturn]

Someone posted another concern on discord:

I think there's a bug in the community CLIP guidance example -

diffusers/examples/community/clip_guided_stable_diffusion.py

Lines 132 to 135 in 8211b62

	pred_original_sample = (latents - beta_prod_t ** (0.5) * noise_pred) / alpha_prod_t ** (0.5)
	fac = torch.sqrt(beta_prod_t)
	sample = pred_original_sample * (fac) + latents * (1 - fac)

IIUC the correct behavior would be to use pred_original_sample to do the CLIP comparison - i.e. it should be the sample, not pred_original_sample * (fac) + latents * (1 - fac)

[patil-suraj]

That's a great catch @keturn ! Sorry to come back to this only now, will send a fix right away.

[patil-suraj]

Also updated the colab notebook https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/CLIP_Guided_Stable_diffusion_with_diffusers.ipynb#scrollTo=-bzXx-2EhlQY

[ajundo]

Someone posted another concern on discord:

I think there's a bug in the community CLIP guidance example -

diffusers/examples/community/clip_guided_stable_diffusion.py

Lines 132 to 135 in 8211b62

	pred_original_sample = (latents - beta_prod_t ** (0.5) * noise_pred) / alpha_prod_t ** (0.5)
	fac = torch.sqrt(beta_prod_t)
	sample = pred_original_sample * (fac) + latents * (1 - fac)

IIUC the correct behavior would be to use pred_original_sample to do the CLIP comparison - i.e. it should be the sample, not pred_original_sample * (fac) + latents * (1 - fac)

Afaik, this is correct because the conditional diffusion is conditional on predicted x_t rather than predicted x_0。

[zzhang2816]

I think the correct format is "sample = pred_original_sample * (fac) + noise_pred * (1 - fac)"