Train Faster R-CNN with negative samples

[mnc537]

Hello,

This PR allows FasterRCNN to train with negative samples.
Related to: #1598

When defining the dataset, one needs to set the field boxes in the target dict as torch.zeros((0, 4), dtype=torch.float32) for negative images since boxes is required.

This is how target should look like:

target = {}
target["boxes"] = torch.zeros((0, 4), dtype=torch.float32)
target["labels"] = torch.zeros((1, 1), dtype=torch.int64)
target["image_id"] = image_id
target["area"] = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
target["iscrowd"] = torch.zeros((0,), dtype=torch.int64)

[cpuhrsch]

I'm out of depth for this particular code-piece and will leave the review to @fmassa

[fmassa]

The PR looks great Monica, thanks a lot!

I have only a few minor comments, can you look into it? Once they are addressed this will be good to merge.

Also, I think that we should also make sure that mask r-cnn and keypoint r-cnn work with empty targets, could you look into it in a follow-up PR?

Once again thanks a lot!

[fmassa]

nit: this conditional is not needed anymore

[fmassa]

Do we need to take the device of gt_boxes_in_image into account?

[fmassa]

I think it would be safer if we also take the device of proposals_in_image into account while creating those tensors

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@d45a77d). Click here to learn what that means.
The diff coverage is 0%.

@@           Coverage Diff            @@
##             master   #1911   +/-   ##
========================================
  Coverage          ?   0.48%           
========================================
  Files             ?      92           
  Lines             ?    7442           
  Branches          ?    1133           
========================================
  Hits              ?      36           
  Misses            ?    7393           
  Partials          ?      13

Impacted Files	Coverage Δ
torchvision/models/detection/roi_heads.py	0% <0%> (ø)
torchvision/models/detection/rpn.py	0% <0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d45a77d...b7ed9ac. Read the comment docs.

[fmassa]

Thanks a lot Monica!

[rronan]

Thanks for this feature!

This is how target should look like:

target = {}
target["boxes"] = torch.zeros((0, 4), dtype=torch.float32)
target["labels"] = torch.zeros((1, 1), dtype=torch.int64)
target["image_id"] = image_id
target["area"] = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
target["iscrowd"] = torch.zeros((0,), dtype=torch.int64)

One question though: why does target["labels"] look like this? Shouldn't it be of length 0 (like boxes), or with only one dimension (not two) like specified here:
https://github.com/pytorch/vision/blob/master/torchvision/models/detection/faster_rcnn.py#L47

[fmassa]

@rronan I agree, I was wondering the same thing. @mnc537 is this a typo in the test?

[mnc537]

@rronan, right! It should be of length 0. There is a typo in the test, @fmassa. I'll fix it.

[fmassa]

Thanks @mnc537 !

[rronan]

Thank you @mnc537. I've been using this PR and did not experience any issue yet.

[Kirayue]

Hi, @fmassa, @mnc537

I have a question about the targe["labels"],

According to the PR,

target = {}
target["boxes"] = torch.zeros((0, 4), dtype=torch.float32)
target["labels"] = torch.zeros((1, 1), dtype=torch.int64)
target["image_id"] = image_id
target["area"] = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
target["iscrowd"] = torch.zeros((0,), dtype=torch.int64)

@mnc537 said there was a typo, so the target should look like
target["labels"] = torch.zeros((1, ), dtype=torch.int64) or
target["labels"] = torch.ones((0, ), dtype=torch.int64) or
target["labels"] = torch.zeros((1, 1), dtype=torch.int64). or
target["labels"] = torch.zeros((0, )), dytype=torch.int64

I tried all above but there weren't any errors.

Thank you for the feature.

[dccf36]

Does anyone encounter the problem that loss explodes after I add the negative background

[samra-irshad]

@mnc537 Thanks for this feature. Just wondering, do we need to increase the number of classes once we add the targets for negative samples?

[Kirayue]

@samra-irshad The number of classes is the number of your objects + 1(negative samples, or image with only background).

https://pytorch.org/vision/stable/models.html#id35

[samra-irshad]

@Kirayue So background class (0) and negative samples (images with no object) should have same label? Or I should allocate an additional label to images with no objects?

[Kirayue]

@samra-irshad, they are the same, so you do not need to add an additional label to indicate no objects.

[ashep29]

This error occurs when I try to use negative sampling on unlabelled images using:

target["boxes"] = torch.zeros((0, 4), dtype=torch.float32)
target["labels"] = torch.zeros((1, 1), dtype=torch.int64)

ValueError: Expected target boxes to be a tensorof shape [N, 4], got torch.Size([0]).

Any ideas on how to address this?

[jodumagpi]

@mnc537 how should the target masks look like??

[carsumptive]

Hello, this is a feature that I wish was elaborated upon a bit further in the docs as it is quite useful and I am trying to get it to work. I believe I can implement it when using the Detecto library but I am having a couple of issues, thought it would be best to follow up on this thread if anyone is watching it.

1. When passing 0,0,0,0 box values into the dataloader, the torchvision library's function in generalizedrcnn stops me for having degenerate boxes because the dimensions are wrong. Any word on working around this? I assume that logic is there for a reason.. This PR just doesn't seem to address that, any help is greatly appreciated.