Skip to content

feat: add functional center crop on mask #5961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

federicopozzi33
Copy link
Contributor

@federicopozzi33 federicopozzi33 commented May 7, 2022

Related to #5782

Code
import torch
import torchvision
from torchvision.prototype import features
from torchvision.prototype.transforms.functional import center_crop_bounding_box, center_crop_image_tensor, center_crop_segmentation_mask

size = (64, 76)
# xyxy format
in_boxes = [
    [10, 15, 25, 35],
    [50, 5, 70, 22],
    [45, 46, 56, 62],
    [size[1] // 2 - 3, size[0] // 2 - 3, size[1] // 2 + 3, size[0] // 2 + 3]
]
labels = [1, 2, 3, 4]

im1 = 255 * np.ones(size + (3, ), dtype=np.uint8)
mask = np.zeros(size, dtype=np.int64)
for in_box, label in zip(in_boxes, labels):
    im1[in_box[1]:in_box[3], in_box[0]:in_box[2], :] = (127, 127, 127)
    mask[in_box[1]:in_box[3], in_box[0]:in_box[2]] = label
    
t_im1 = torch.tensor(im1).permute(2, 0, 1).view(1, 3, *size)

in_boxes = features.BoundingBox(
    in_boxes, format=features.BoundingBoxFormat.XYXY, image_size=size
)
in_mask = features.SegmentationMask(torch.tensor(mask)).view(1, *size)
    
output_size = (45, 50)

out_boxes = center_crop_bounding_box(
    in_boxes, 
    in_boxes.format,
    output_size=output_size,
    image_size=in_boxes.image_size
)
print(out_boxes)

out_mask = center_crop_segmentation_mask(
    in_mask,  
    output_size 
)

t_im2 = center_crop_image_tensor(t_im1, output_size)


import cv2
import matplotlib.pyplot as plt


plt.figure(figsize=(14, 10))

plt.subplot(2,3,1)
plt.title("Input image + bboxes")
r1 = t_im1[0, ...].permute(1, 2, 0).contiguous().cpu().numpy()
for in_box in in_boxes:    
    r1 = cv2.rectangle(r1, (in_box[0].item(), in_box[1].item()), (in_box[2].item(), in_box[3].item()), (255, 127, 0))
plt.imshow(r1)


plt.subplot(2,3,2)
plt.title("Input segm mask")
plt.imshow(in_mask[0, :, :].cpu().numpy())


plt.subplot(2,3,3)
plt.title("Input image + bboxes + segm mask")
plt.imshow(r1, alpha=0.5)
plt.imshow(in_mask[0, :, :].cpu().numpy(), alpha=0.75)


plt.subplot(2,3,4)
plt.title("Output image + bboxes")
r2 = t_im2[0, ...].permute(1, 2, 0).contiguous().cpu().numpy()
for out_box in out_boxes:
    out_box = np.round(out_box.cpu().numpy()).astype("int32")
    r2 = cv2.rectangle(r2, (out_box[0], out_box[1]), (out_box[2], out_box[3]), (255, 127, 0), 0)
plt.imshow(r2)


plt.subplot(2,3,5)
plt.title("Output segm mask")
plt.imshow(out_mask[0, :, :].cpu().numpy())

plt.subplot(2,3,6)
plt.title("Output image + bboxes + segm mask")
plt.imshow(r2, alpha=0.5)
plt.imshow(out_mask[0, :, :].cpu().numpy(), alpha=0.75)

pad_viz

@federicopozzi33
Copy link
Contributor Author

@pmeier, @vfdev-5 thoughts about testing the correctness of the function by mocking the function that really does the work? It's a proposal related to what we were saying in #5866.

IMO, there is no logic in center_crop_segmentation_mask, I'm just calling the center_crop_image_tensor function. If the latter is properly tested, the current test should be enough.

@vfdev-5, for the visualization side, the function center_crop_bounding_box is still missing. Is it ok to update the description as soon as it's implemented?

@vfdev-5
Copy link
Collaborator

vfdev-5 commented May 8, 2022

IMO, there is no logic in center_crop_segmentation_mask, I'm just calling the center_crop_image_tensor function. If the latter is properly tested, the current test should be enough.

@federicopozzi33 I agree, I have a similar reflection about that while working on center_crop_bounding_box. Currently, I opted to write compute_expected_bbox function and check for results... There could be however few corner cases.

@pmeier, @vfdev-5 thoughts about testing the correctness of the function by mocking the function that really does the work? It's a proposal related to what we were saying in #5866.

We have to think about how to test composed ops in general. 1) Ensure that we call base ops in a right way or 2) check if the output is what we expect (expected result could be either hard-coded, computed using base ops or recomputed in a different way) or 3) something else ?

@vfdev-5, for the visualization side, the function center_crop_bounding_box is still missing. Is it ok to update the description as soon as it's implemented?

You can make visualization for segm mask and then add bbox when available ?

@vfdev-5
Copy link
Collaborator

vfdev-5 commented May 9, 2022

@federicopozzi33 fyi center crop for bbox PR: #5972

@federicopozzi33
Copy link
Contributor Author

@federicopozzi33 I agree, I have a similar reflection about that while working on center_crop_bounding_box. Currently, I opted to write compute_expected_bbox function and check for results... There could be however few corner cases.

I added a test with random input too, but IMO it's not needed here: I don't see any corner cases.

We have to think about how to test composed ops in general. 1) Ensure that we call base ops in a right way or 2) check if the output is what we expect (expected result could be either hard-coded, computed using base ops or recomputed in a different way) or 3) something else ?

We can do both 1) and 2). I'd achieve the former by patching the base ops, and the latter with fixed input.
Then, 3) specific test cases can be added for corner cases.

You can make visualization for segm mask and then add bbox when available ?

I've updated the description.

@federicopozzi33 federicopozzi33 marked this pull request as ready for review May 10, 2022 19:54
@vfdev-5
Copy link
Collaborator

vfdev-5 commented May 11, 2022

@vfdev-5
Copy link
Collaborator

vfdev-5 commented May 11, 2022

@federicopozzi33 thanks for the update! Code and test looks good to me (I left a minor suggestion). Once you fixed the problem with CI (probably due to resolving conflicts) we can merge it.

@pmeier any thoughts from your side about testing, particularly mocking part ?

Copy link
Collaborator

@pmeier pmeier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor nit about mocking in pytest. Otherwise testing looks good. Still, I want to hear @NicolasHug's opinion on mocking in general for this use case.

Copy link
Collaborator

@pmeier pmeier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've discussed this with @NicolasHug offline and we both agree the mock test seems a little overkill here. The function is a one-liner without any additional logic. The test is orders of magnitude more complex. Thus, in the future it is more likely that the test breaks rather then the kernel.

Plus, we already have tests that check the numerical correctness. Thus, the test is not bringing any new information.

What was the reason to go for the complex mocking here?

@vfdev-5
Copy link
Collaborator

vfdev-5 commented May 11, 2022

What was the reason to go for the complex mocking here?

It is the first time we opt to use mock. I agree that in case of mask it is one-liner and it does not bring any useful checks but it can serve as an example for other composite ops, IMO.
Anyway, as you are reworking tests there will be other pass on tests, so no strong opinion from my side about removing or keeping this test.

@federicopozzi33
Copy link
Contributor Author

federicopozzi33 commented May 12, 2022

I've discussed this with @NicolasHug offline and we both agree the mock test seems a little overkill here. The function is a one-liner without any additional logic. The test is orders of magnitude more complex. Thus, in the future it is more likely that the test breaks rather then the kernel.

Plus, we already have tests that check the numerical correctness. Thus, the test is not bringing any new information.

What was the reason to go for the complex mocking here?

As I said in some previous comment, this was just a proposal, and that can be useful in some cases, such as in #5866 (BTW, do you think that missing test cases can be covered using mocking? @vfdev-5).

I agree with you that the function is very simple, but the function called inside is not straightforward to test.
For instance, some test cases such as - output_size bigger than original tensor size - are missing.
Covering all test cases means writing a _prepare_expected function which implements the whole center_crop_image_tensor logic.

The purpose of mocking here is simple: center_crop_segmentation_mask is just a sort of wrapper around center_crop_image_tensor. center_crop_image_tensor is already tested.
So, all I need to do is to check that:

  • the inner function is called with the expected arguments
  • the return value of inner function is the return value of the target function (center_crop_segmentation_mask).

Anyway, I not sure if I did right removing the test with mocking.

@federicopozzi33 federicopozzi33 requested review from pmeier and vfdev-5 May 12, 2022 19:37
@pmeier
Copy link
Collaborator

pmeier commented May 13, 2022

@federicopozzi33 I agree with the general sentiment, but I would go a different route. If center_crop_image_tensor is already tested and thus implements a _prepare_expected function for all cases, can't we simply re-use it here?

@federicopozzi33
Copy link
Contributor Author

federicopozzi33 commented May 13, 2022

@federicopozzi33 I agree with the general sentiment, but I would go a different route. If center_crop_image_tensor is already tested and thus implements a _prepare_expected function for all cases, can't we simply re-use it here?

If the only thing that my function does is to call another tested function, IMO, mocking is the way. It's the same that calling the inner function, but without really doing it (because I'm really interested in the result). Just two different ways to approach the same problem. You choose :)

So, should I add replace my naive prepare_expected to cover many more test cases?

@pmeier
Copy link
Collaborator

pmeier commented May 13, 2022

If the only thing that my function does is to call another tested function, IMO, mocking is the way.

That is a dangerous conclusion. Mocking something always means that we switch from black box testing to white box testing. Meaning, we now need to know the internals of the function we want to test. That isn't a bad thing in general, but brings other problems with it. For example, if the internals change but the interface stays the same, the test might break although it shouldn't.

Imagine we rename center_crop_image_tensor to center_crop_image. The test for the segmentation mask would now fail since the mock target needs to change. Admittedly, the fix wouldn't be hard, but it is still a breakage that does not need to happen.

In this situation here, IMO it would be best to just use the expected result computation for images and segmentation and stick to black box testing.

@federicopozzi33 federicopozzi33 force-pushed the feature/5782-proto-center-crop-mask branch from cb3161f to b6953b2 Compare May 14, 2022 08:19
@federicopozzi33
Copy link
Contributor Author

If the only thing that my function does is to call another tested function, IMO, mocking is the way.

That is a dangerous conclusion. Mocking something always means that we switch from black box testing to white box testing. Meaning, we now need to know the internals of the function we want to test. That isn't a bad thing in general, but brings other problems with it. For example, if the internals change but the interface stays the same, the test might break although it shouldn't.

Imagine we rename center_crop_image_tensor to center_crop_image. The test for the segmentation mask would now fail since the mock target needs to change. Admittedly, the fix wouldn't be hard, but it is still a breakage that does not need to happen.

In this situation here, IMO it would be best to just use the expected result computation for images and segmentation and stick to black box testing.

I agree with you. I'd just like to clarify one point: IMO, mocking a public function is very different from mocking a private one, especially in cases like ours, considering that modifying public functions means doing a "major change".

Anyway, I added some extra test cases (output size bigger than image and odd output dimensions) and I'm computing the expected with the center_crop_image_tensor, so please take a look.

@pmeier
Copy link
Collaborator

pmeier commented May 16, 2022

considering that modifying public functions means doing a "major change".

That is true in general, but not so much during the prototype phase.

Comment on lines 426 to 427
make_segmentation_masks(),
[[4, 3], [42, 70], [4]], # crop sizes < image sizes, crop_sizes > image sizes, single crop size
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we maybe not rely on the default values here to make sure we actually get the cases detailed in the comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still use the default values, but I have made them explicit.

@pytest.mark.parametrize("output_size", [[4, 3], [4], [7, 7]])
def test_correctness_center_crop_segmentation_mask(device, output_size):
def _compute_expected_segmentation_mask(mask, output_size):
return F.center_crop_image_tensor(mask, output_size)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really happy about this given that we don't seem to have a reference test that makes sure F.center_crop_image_tensor. Thus, if that had a bug, we would propagate it to this test.

Plus, we now use exactly the same implementation for the test and kernel and thus are unable to detect any bug with this test.

Maybe we can have something like

class TestCorrectnessCenterCrop:
    def _compute_expected_image_or_segmentation_mask(self, input, output_size):
        # manual implementation that does not use our tensor kernels
        # IMO, it would be fine to use `center_crop_image_pil` here.
        pass
    
    def test_image(self):
        pass
    
    def test_segmentation_mask(self):
        pass

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the problem here is with the _compute_expected_segmentation_mask implementation which is exactly the same as F.center_crop_segmentation_mask. IMO, this way of testing does not bring any value.

As for restructuring with TestCorrectnessCenterCrop, @pmeier it can be maybe done in your functional tests refactor PR, not here.

Let's move forward with this. If we abandoned somehow the idea of mocking F.center_crop_image_tensor op and just checking the args, now let's recode _compute_expected_segmentation_mask such that we compute in another way a center crop and check output vs expected results.

@federicopozzi33
Copy link
Contributor Author

Sorry, I'm pretty busy these days, I'll fix the PR in the next few days (probably during the weekend).

@federicopozzi33 federicopozzi33 force-pushed the feature/5782-proto-center-crop-mask branch from c939a5f to 802b9b4 Compare May 21, 2022 08:02
@federicopozzi33 federicopozzi33 requested a review from pmeier May 21, 2022 08:03
Copy link
Collaborator

@vfdev-5 vfdev-5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @federicopozzi33

Copy link
Collaborator

@pmeier pmeier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +1361 to +1362
left = round((image_width - crop_width) * 0.5)
top = round((image_height - crop_height) * 0.5)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The kernel has an additional int call:

crop_top = int(round((image_height - crop_height) / 2.0))
crop_left = int(round((image_width - crop_width) / 2.0))

@vfdev-5 I recall we had issues with round before. Should we just switch to int in general?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pmeier I do not quite remember what was the issue with round (maybe jit behaves differently to eager mode). For me the code you mention is more like a definition of crop_top and crop_left. For example, for bboxes we could also keep these values as float but let's define that crop_top/left are rounded integers.

@vfdev-5 vfdev-5 merged commit 3a2631b into pytorch:main May 23, 2022
facebook-github-bot pushed a commit that referenced this pull request Jun 1, 2022
Summary:
* feat: add functional center crop on mask

* test: add correctness center crop with random segmentation mask

* test: improvements

* test: improvements

* Apply suggestions from code review

Reviewed By: NicolasHug

Differential Revision: D36760924

fbshipit-source-id: 2e033926b18a34676a367331a03e45ca2ffd9cdc

Co-authored-by: Philip Meier <[email protected]>
Co-authored-by: Federico Pozzi <[email protected]>
Co-authored-by: vfdev <[email protected]>
Co-authored-by: Philip Meier <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants