feat: add functional center crop on mask

[federicopozzi33]

Related to #5782

Code

import torch
import torchvision
from torchvision.prototype import features
from torchvision.prototype.transforms.functional import center_crop_bounding_box, center_crop_image_tensor, center_crop_segmentation_mask

size = (64, 76)
# xyxy format
in_boxes = [
    [10, 15, 25, 35],
    [50, 5, 70, 22],
    [45, 46, 56, 62],
    [size[1] // 2 - 3, size[0] // 2 - 3, size[1] // 2 + 3, size[0] // 2 + 3]
]
labels = [1, 2, 3, 4]

im1 = 255 * np.ones(size + (3, ), dtype=np.uint8)
mask = np.zeros(size, dtype=np.int64)
for in_box, label in zip(in_boxes, labels):
    im1[in_box[1]:in_box[3], in_box[0]:in_box[2], :] = (127, 127, 127)
    mask[in_box[1]:in_box[3], in_box[0]:in_box[2]] = label
    
t_im1 = torch.tensor(im1).permute(2, 0, 1).view(1, 3, *size)

in_boxes = features.BoundingBox(
    in_boxes, format=features.BoundingBoxFormat.XYXY, image_size=size
)
in_mask = features.SegmentationMask(torch.tensor(mask)).view(1, *size)
    
output_size = (45, 50)

out_boxes = center_crop_bounding_box(
    in_boxes, 
    in_boxes.format,
    output_size=output_size,
    image_size=in_boxes.image_size
)
print(out_boxes)

out_mask = center_crop_segmentation_mask(
    in_mask,  
    output_size 
)

t_im2 = center_crop_image_tensor(t_im1, output_size)


import cv2
import matplotlib.pyplot as plt


plt.figure(figsize=(14, 10))

plt.subplot(2,3,1)
plt.title("Input image + bboxes")
r1 = t_im1[0, ...].permute(1, 2, 0).contiguous().cpu().numpy()
for in_box in in_boxes:    
    r1 = cv2.rectangle(r1, (in_box[0].item(), in_box[1].item()), (in_box[2].item(), in_box[3].item()), (255, 127, 0))
plt.imshow(r1)


plt.subplot(2,3,2)
plt.title("Input segm mask")
plt.imshow(in_mask[0, :, :].cpu().numpy())


plt.subplot(2,3,3)
plt.title("Input image + bboxes + segm mask")
plt.imshow(r1, alpha=0.5)
plt.imshow(in_mask[0, :, :].cpu().numpy(), alpha=0.75)


plt.subplot(2,3,4)
plt.title("Output image + bboxes")
r2 = t_im2[0, ...].permute(1, 2, 0).contiguous().cpu().numpy()
for out_box in out_boxes:
    out_box = np.round(out_box.cpu().numpy()).astype("int32")
    r2 = cv2.rectangle(r2, (out_box[0], out_box[1]), (out_box[2], out_box[3]), (255, 127, 0), 0)
plt.imshow(r2)


plt.subplot(2,3,5)
plt.title("Output segm mask")
plt.imshow(out_mask[0, :, :].cpu().numpy())

plt.subplot(2,3,6)
plt.title("Output image + bboxes + segm mask")
plt.imshow(r2, alpha=0.5)
plt.imshow(out_mask[0, :, :].cpu().numpy(), alpha=0.75)

[federicopozzi33]

@pmeier, @vfdev-5 thoughts about testing the correctness of the function by mocking the function that really does the work? It's a proposal related to what we were saying in #5866.

IMO, there is no logic in center_crop_segmentation_mask, I'm just calling the center_crop_image_tensor function. If the latter is properly tested, the current test should be enough.

@vfdev-5, for the visualization side, the function center_crop_bounding_box is still missing. Is it ok to update the description as soon as it's implemented?

[vfdev-5]

IMO, there is no logic in center_crop_segmentation_mask, I'm just calling the center_crop_image_tensor function. If the latter is properly tested, the current test should be enough.

@federicopozzi33 I agree, I have a similar reflection about that while working on center_crop_bounding_box. Currently, I opted to write compute_expected_bbox function and check for results... There could be however few corner cases.

@pmeier, @vfdev-5 thoughts about testing the correctness of the function by mocking the function that really does the work? It's a proposal related to what we were saying in #5866.

We have to think about how to test composed ops in general. 1) Ensure that we call base ops in a right way or 2) check if the output is what we expect (expected result could be either hard-coded, computed using base ops or recomputed in a different way) or 3) something else ?

@vfdev-5, for the visualization side, the function center_crop_bounding_box is still missing. Is it ok to update the description as soon as it's implemented?

You can make visualization for segm mask and then add bbox when available ?

[vfdev-5]

@federicopozzi33 fyi center crop for bbox PR: #5972

[federicopozzi33]

@federicopozzi33 I agree, I have a similar reflection about that while working on center_crop_bounding_box. Currently, I opted to write compute_expected_bbox function and check for results... There could be however few corner cases.

I added a test with random input too, but IMO it's not needed here: I don't see any corner cases.

We have to think about how to test composed ops in general. 1) Ensure that we call base ops in a right way or 2) check if the output is what we expect (expected result could be either hard-coded, computed using base ops or recomputed in a different way) or 3) something else ?

We can do both 1) and 2). I'd achieve the former by patching the base ops, and the latter with fixed input.
Then, 3) specific test cases can be added for corner cases.

You can make visualization for segm mask and then add bbox when available ?

I've updated the description.

[vfdev-5]

@federicopozzi33 CI failures are real : https://github.com/pytorch/vision/runs/6376638110?check_suite_focus=true#step:9:127
please check

[vfdev-5]

@federicopozzi33 thanks for the update! Code and test looks good to me (I left a minor suggestion). Once you fixed the problem with CI (probably due to resolving conflicts) we can merge it.

@pmeier any thoughts from your side about testing, particularly mocking part ?

[pmeier]

One minor nit about mocking in pytest. Otherwise testing looks good. Still, I want to hear @NicolasHug's opinion on mocking in general for this use case.

[pmeier]

I've discussed this with @NicolasHug offline and we both agree the mock test seems a little overkill here. The function is a one-liner without any additional logic. The test is orders of magnitude more complex. Thus, in the future it is more likely that the test breaks rather then the kernel.

Plus, we already have tests that check the numerical correctness. Thus, the test is not bringing any new information.

What was the reason to go for the complex mocking here?

[vfdev-5]

What was the reason to go for the complex mocking here?

It is the first time we opt to use mock. I agree that in case of mask it is one-liner and it does not bring any useful checks but it can serve as an example for other composite ops, IMO.
Anyway, as you are reworking tests there will be other pass on tests, so no strong opinion from my side about removing or keeping this test.

[federicopozzi33]

I've discussed this with @NicolasHug offline and we both agree the mock test seems a little overkill here. The function is a one-liner without any additional logic. The test is orders of magnitude more complex. Thus, in the future it is more likely that the test breaks rather then the kernel.

Plus, we already have tests that check the numerical correctness. Thus, the test is not bringing any new information.

What was the reason to go for the complex mocking here?

As I said in some previous comment, this was just a proposal, and that can be useful in some cases, such as in #5866 (BTW, do you think that missing test cases can be covered using mocking? @vfdev-5).

I agree with you that the function is very simple, but the function called inside is not straightforward to test.
For instance, some test cases such as - output_size bigger than original tensor size - are missing.
Covering all test cases means writing a _prepare_expected function which implements the whole center_crop_image_tensor logic.

The purpose of mocking here is simple: center_crop_segmentation_mask is just a sort of wrapper around center_crop_image_tensor. center_crop_image_tensor is already tested.
So, all I need to do is to check that:

• the inner function is called with the expected arguments
• the return value of inner function is the return value of the target function (center_crop_segmentation_mask).

Anyway, I not sure if I did right removing the test with mocking.

[pmeier]

@federicopozzi33 I agree with the general sentiment, but I would go a different route. If center_crop_image_tensor is already tested and thus implements a _prepare_expected function for all cases, can't we simply re-use it here?

[federicopozzi33]

@federicopozzi33 I agree with the general sentiment, but I would go a different route. If center_crop_image_tensor is already tested and thus implements a _prepare_expected function for all cases, can't we simply re-use it here?

If the only thing that my function does is to call another tested function, IMO, mocking is the way. It's the same that calling the inner function, but without really doing it (because I'm really interested in the result). Just two different ways to approach the same problem. You choose :)

So, should I add replace my naive prepare_expected to cover many more test cases?

[pmeier]

If the only thing that my function does is to call another tested function, IMO, mocking is the way.

That is a dangerous conclusion. Mocking something always means that we switch from black box testing to white box testing. Meaning, we now need to know the internals of the function we want to test. That isn't a bad thing in general, but brings other problems with it. For example, if the internals change but the interface stays the same, the test might break although it shouldn't.

Imagine we rename center_crop_image_tensor to center_crop_image. The test for the segmentation mask would now fail since the mock target needs to change. Admittedly, the fix wouldn't be hard, but it is still a breakage that does not need to happen.

In this situation here, IMO it would be best to just use the expected result computation for images and segmentation and stick to black box testing.

[federicopozzi33]

If the only thing that my function does is to call another tested function, IMO, mocking is the way.

That is a dangerous conclusion. Mocking something always means that we switch from black box testing to white box testing. Meaning, we now need to know the internals of the function we want to test. That isn't a bad thing in general, but brings other problems with it. For example, if the internals change but the interface stays the same, the test might break although it shouldn't.

Imagine we rename center_crop_image_tensor to center_crop_image. The test for the segmentation mask would now fail since the mock target needs to change. Admittedly, the fix wouldn't be hard, but it is still a breakage that does not need to happen.

In this situation here, IMO it would be best to just use the expected result computation for images and segmentation and stick to black box testing.

I agree with you. I'd just like to clarify one point: IMO, mocking a public function is very different from mocking a private one, especially in cases like ours, considering that modifying public functions means doing a "major change".

Anyway, I added some extra test cases (output size bigger than image and odd output dimensions) and I'm computing the expected with the center_crop_image_tensor, so please take a look.

[pmeier]

considering that modifying public functions means doing a "major change".

That is true in general, but not so much during the prototype phase.

[pmeier]

Should we maybe not rely on the default values here to make sure we actually get the cases detailed in the comment?

[federicopozzi33]

I still use the default values, but I have made them explicit.

[pmeier]

I'm not really happy about this given that we don't seem to have a reference test that makes sure F.center_crop_image_tensor. Thus, if that had a bug, we would propagate it to this test.

Plus, we now use exactly the same implementation for the test and kernel and thus are unable to detect any bug with this test.

Maybe we can have something like

class TestCorrectnessCenterCrop:
    def _compute_expected_image_or_segmentation_mask(self, input, output_size):
        # manual implementation that does not use our tensor kernels
        # IMO, it would be fine to use `center_crop_image_pil` here.
        pass
    
    def test_image(self):
        pass
    
    def test_segmentation_mask(self):
        pass

[vfdev-5]

I think the problem here is with the _compute_expected_segmentation_mask implementation which is exactly the same as F.center_crop_segmentation_mask. IMO, this way of testing does not bring any value.

As for restructuring with TestCorrectnessCenterCrop, @pmeier it can be maybe done in your functional tests refactor PR, not here.

Let's move forward with this. If we abandoned somehow the idea of mocking F.center_crop_image_tensor op and just checking the args, now let's recode _compute_expected_segmentation_mask such that we compute in another way a center crop and check output vs expected results.

[federicopozzi33]

Sorry, I'm pretty busy these days, I'll fix the PR in the next few days (probably during the weekend).

[vfdev-5]

LGTM, thanks @federicopozzi33

[pmeier]

Thanks @federicopozzi33!

[pmeier]

The kernel has an additional int call:

vision/torchvision/prototype/transforms/functional/_geometry.py

Lines 585 to 586 in b969cca

	crop_top = int(round((image_height - crop_height) / 2.0))
	crop_left = int(round((image_width - crop_width) / 2.0))

@vfdev-5 I recall we had issues with round before. Should we just switch to int in general?

[vfdev-5]

@pmeier I do not quite remember what was the issue with round (maybe jit behaves differently to eager mode). For me the code you mention is more like a definition of crop_top and crop_left. For example, for bboxes we could also keep these values as float but let's define that crop_top/left are rounded integers.