Running revision="fp16", torch_dtype=torch.float16 on mps M1

[MatthewWaller]

Describe the bug

I'm using the following code:

!pip install diffusers
!pip install transformers scipy ftfy

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16, use_auth_token=True)

prompt = "Pineapple on a white table"
pipe.to("mps")

# Found docs suggested this to get around a bug
with autocast('mps'):
    _ = pipe(prompt, num_inference_steps=1)

with autocast('mps'):
    image = pipe(prompt).images[0]
image.save("hmmm.png")

But I always get the error: RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

Is there a way around this error? Seems like the values aren't getting cast right, or this layer isn't getting the message.

Reproduction

!pip install diffusers
!pip install transformers scipy ftfy

pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16, use_auth_token=True)

prompt = "Very good prompt"
pipe.to("mps")

# Found docs suggested this to get around a bug
with autocast('mps'):
    _ = pipe(prompt, num_inference_steps=1)

with autocast('mps'):
    image = pipe(prompt).images[0]
image.save("hmmm.png")

Logs

Traceback (most recent call last):
  File "/Users/administrator/Desktop/stable_diffusion/main.py", line 258, in <module>
    image = pipe(prompt, num_inference_steps=21, latents=latents).images[0]
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 182, in __call__
    text_embeddings = self.text_encoder(text_input.input_ids.to(self.device))[0]
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py", line 731, in forward
    return self.text_model(
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py", line 649, in forward
    encoder_outputs = self.encoder(
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py", line 578, in forward
    layer_outputs = encoder_layer(
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py", line 320, in forward
    hidden_states = self.layer_norm1(hidden_states)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 190, in forward
    return F.layer_norm(
  File "/opt/anaconda3/envs/more_stable/lib/python3.8/site-packages/torch/nn/functional.py", line 2511, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

System Info

python 3.9
torch==1.13.0.dev20220917
diffusers==0.3.0