Check tolerances for prototype transforms kernel reference tests

[pmeier]

Restoring the tolerances to

vision/test/prototype_transforms_kernel_infos.py

Lines 64 to 67 in f32600b

	DEFAULT_PIL_REFERENCE_CLOSENESS_KWARGS = {
	(("TestKernels", "test_against_reference"), torch.float32, "cpu"): dict(atol=1e-5, rtol=0, agg_method="mean"),
	(("TestKernels", "test_against_reference"), torch.uint8, "cpu"): dict(atol=1e-5, rtol=0, agg_method="mean"),
	}

that will be cranked up by #6934 surfaces failures on the following kernels in the reference test:

• adjust_brightness_image_tensor
• adjust_contrast_image_tensor
• adjust_gamma_image_tensor
• adjust_hue_image_tensor
• adjust_saturation_image_tensor
• adjust_sharpness_image_tensor
• affine_image_tensor
• affine_mask
• autocontrast_image_tensor
• center_crop_image_tensor
• crop_image_tensor
• elastic_image_tensor
• elastic_mask
• equalize_image_tensor
• five_crop_image_tensor
• horizontal_flip_image_tensor
• invert_image_tensor
• pad_image_tensor
• perspective_image_tensor
• perspective_mask
• posterize_image_tensor
• resize_image_tensor
• resize_mask
• resized_crop_image_tensor
• resized_crop_mask
• rotate_image_tensor
• rotate_mask
• solarize_image_tensor
• ten_crop_image_tensor
• vertical_flip_image_tensor

We need to go through each of them to see if there is a systematic problem here. However, this is not reason to "panic" since

1. these kernels are used in the training pipeline and we are seeing no accuracy drop compared to v1
2. these kernels are also tested for consistency against v1 and there everything is fine.

cc @vfdev-5 @datumbox @bjuncek

[pmeier]

It seems adjust_hue also has quite the difference on v1:

import torch
from torchvision.transforms import functional as F

torch.manual_seed(0)
image = torch.randint(0, 256, (3, 512, 512), dtype=torch.uint8)

actual = F.adjust_hue(image, hue_factor=0.1)

expected = F.pil_to_tensor(F.adjust_hue(F.to_pil_image(image), hue_factor=0.1))

torch.testing.assert_close(actual, expected)

AssertionError: Tensor-likes are not equal!

Mismatched elements: 335568 / 786432 (42.7%)
Greatest absolute difference: 15 at index (1, 110, 148)
Greatest relative difference: inf at index (0, 31, 485)

[pmeier]

All the remaining kernels that exhibit these large tolerances, internally use interpolation. Unfortunately for us, the PyTorch ops don't match the interpolation behavior of PIL. Consider the following setup:

import PIL.Image
import PIL.ImageDraw

from torchvision.prototype.transforms import functional as F, InterpolationMode

image_pil = PIL.Image.new("RGB", (256, 256))
draw = PIL.ImageDraw.Draw(image_pil)
draw.ellipse((64, 96, 192, 160), fill=(255, 255, 255))

image_tensor = F.to_image_tensor(image_pil)

size = (233, 147)
abs_diff_images = []
for interpolation in [
    InterpolationMode.NEAREST,
    InterpolationMode.NEAREST_EXACT,
    InterpolationMode.BILINEAR,
    InterpolationMode.BICUBIC,
]:
    resized_image_pil = F.resize(image_pil, size, interpolation=interpolation)
    resized_image_tensor = F.resize(
        image_tensor,
        size,
        interpolation=interpolation,
        antialias=interpolation in {InterpolationMode.BILINEAR, InterpolationMode.BICUBIC},
    )

    abs_diff_image = resized_image_tensor.sub(F.to_image_tensor(resized_image_pil)).abs_()

    print(f"MAE for {interpolation.name}: {float(abs_diff_image.float().mean()):.1f}")

    abs_diff_images.append(abs_diff_image)

This gives this input

and the following outputs (absolute difference between tensor and PIL backend for different interpolation modes):

MAE for NEAREST: 0.3
MAE for NEAREST_EXACT: 0.0
MAE for BILINEAR: 0.1
MAE for BICUBIC: 0.7

Except for NEAREST_EXACT (second column) there is a slight deviation around the border of the object. In general though, the kernels are comparable.

However, this is not what we do in our test suite. There the generation logic looks more like

import torch

torch.manual_seed(0)
image_tensor = torch.randint(0, 256, (3, 512, 512), dtype=torch.uint8)
image_pil = F.to_image_pil(image_tensor)

With this we get this input

and the following outputs:

MAE for NEAREST: 119.8
MAE for NEAREST_EXACT: 0.5
MAE for BILINEAR: 24.5
MAE for BICUBIC: 26.6

Even EXACT_NEAREST, which is supposed to aligned with PIL samples one row differently. All other interpolation modes give a completely different result.

This is the reason we are seeing this crazy required tolerances: the difference is basically another random image.

I thought of five ways to move forward in decreasing order of my preference:

1. Change the input generation for the interpolation ops to objects like the ellipse above rather than random images. Together with agg_method="mean" (with that, atol becomes the MAE) we should get reasonable small differences again. Note that we cannot just set the atol, since the difference in the images above is most of the times (and for the NEAREST always) 255.
2. Remove the automatic reference tests for these kernels and replace them with handcrafted ones. This will likely mean we lose some "coverage" since we probably don't want to implement an exhaustive parameter tests manually.
3. Remove NEAREST from the reference tests. This should allow us to use significantly, albeit still high, tolerances.
4. Remove the reference tests for these kernels. We have verified now that the ops indeed work correctly and unless we touch the core logic, they will continue to do that. Of course this is not safe, since sometimes even innocuous looking changes can have an effect. Plus, there might also be changes to the ops in PyTorch core that affect us, but we never know.
5. Keep the test suite as is. With these high tolerances we have basically have option 4., but with extra flakiness and wasted resources sprinkled in.

Thoughts?

[pmeier]

I've played with option 1 from above. Using an ellipse as object that leaves roughly 10% border on each side, we only need to allow an MAE of 1 for resize_image_tensor.

[pmeier]

For elastic_image_tensor there is no more difference at all. With these results I'm now strongly in favor of 1.

[vfdev-5]

For elastic_image_tensor there is no more difference at all. With these results I'm now strongly in favor of 1.

There is no PIL implementation for elastic, we convert PIL to tensor :) That's why there is no diff.

vision/torchvision/transforms/functional.py

Lines 1556 to 1559 in bfb474b

	if not isinstance(img, torch.Tensor):
	if not F_pil._is_pil_image(img):
	raise TypeError(f"img should be PIL Image or Tensor. Got {type(img)}")
	t_img = pil_to_tensor(img)

[pmeier]

Oh boy, good catch 😆 Let me try affine_image_tensor then. We should probably delete the reference test for elastic_image_tensor then. It gives us a false sense of security and I walked right into this pitfall.

[vfdev-5]

We should probably delete the reference test for elastic_image_tensor then. It gives us a false sense of security and I walked right into this pitfall.

Sure. There are few others doing a similar trick, like gaussian blur

[pmeier]

For perspective_image_tensor the required MAE is 2. My point stands 🙂