Skip to content

StableDiffusionDepth2ImgPipeline can't be used with numpy inputs and transformers-4.27 #2786

@antoche

Description

@antoche

Describe the bug

Like most other pipelines, StableDiffusionDepth2ImgPipeline can take either a numpy array or a PIL image as input image.

In diffusers, when using a numpy array, inputs are expected to be in the [-1, 1] range.

In prepare_depth_map, StableDiffusionDepth2ImgPipeline passes the input image to self.feature_extractor:

pixel_values = self.feature_extractor(images=image, return_tensors="pt").pixel_values

As of transformers-4.27, that calls raises an exception when passed a numpy image in the range [-1,1] (the same input did not raise an exception with transformers-4.26, however it's not clear to me whether it was a valid input to the image processor):

ValueError: The image to be converted to a PIL image contains values outside the range [0, 1], got [-1.0, 1.0] which cannot be converted to uint8.

The issue can be fixed by remapping the input given to self.feature_extractor to [0, 1] before making the call. But it's not entirely clear to me whether it should be up to diffusers to make this conversion, or if that is the responsibility of DPTFeatureExtractor, or if a [-1,1] input to DPTImageProcessor is acceptable and it is simply a bug in transformers.

Reproduction

Run StableDiffusionDepth2ImgPipeline with a [-1,1] numpy image.

Logs

No response

System Info

  • diffusers version: 0.14.0
  • Platform: Linux-4.14.240
  • Python version: 3.9.10
  • PyTorch version (GPU?): 1.13.0a0+git49444c3 (True)
  • Huggingface_hub version: 0.13.1
  • Transformers version: 4.27.2
  • Accelerate version: 0.17.1
  • xFormers version: not installed

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions