Finally fix the image-based SD tests #509
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
| return unittest.skipUnless(_run_slow_tests, "test is slow")(test_case) | ||
|
|
||
|
|
||
| def load_image(image: Union[str, PIL.Image.Image]) -> PIL.Image.Image: |
There was a problem hiding this comment.
nice good idea for testing
tests/test_pipelines.py
Outdated
| output_image = ds["output"]["image"][0].resize((768, 512)) | ||
| assert sampled_array.shape == (512, 512, 3) | ||
| # using the mean absolute error due to slightly inconsistent outputs across different GPUs | ||
| assert np.mean(np.abs(sampled_array - expected_array)) < 2.0 |
There was a problem hiding this comment.
2.0 seems to large to me here as discussed. Can we see how big the difference is for pure numpy arrays without any optimization?
Will run:
model_id = "CompVis/stable-diffusion-v1-4"
pipe = StableDiffusionPipeline.from_pretrained(
model_id,
use_auth_token=True,
)
pipe.to(torch_device)
prompt = "A fantasy landscape, trending on artstation"
generator = torch.Generator(device=torch_device).manual_seed(0)
output = pipe(prompt=prompt, guidance_scale=7.5, generator=generator, output_type="np")from current master @anton-l could you do the same and then we compare numpy results across vorace, colab and brutasse
There was a problem hiding this comment.
Note that I just merged @patil-suraj improved attention slicing when disabled, so let's try to include this PR when running the tests
tests/test_pipelines.py
Outdated
| assert sampled_array.shape == (512, 768, 3) | ||
| assert np.max(np.abs(sampled_array - expected_array)) < 1e-4 | ||
| # using the mean absolute error due to slightly inconsistent outputs across different GPUs | ||
| assert np.mean(np.abs(sampled_array - expected_array)) < 2.0 |
There was a problem hiding this comment.
As discussed this can be updated now to much lower tolerance :-)
patil-suraj
left a comment
patil-suraj
left a comment
There was a problem hiding this comment.
LGTM! Same comments as Patrick's
|
Without |
|
Thanks for fixing! |
* Finally fix the image-based SD tests * Remove autocast * Remove autocast in image tests
Solving both the datasets caching and the GPU inconsistency issues