Do you support batch inference?

[zhaoxuyan]

🚀 Feature

How to easily take advantage of batch processing during inference?

Motivation

facebookresearch/Detectron#84

[ppwwyyxx]

You just need to call model with a batch of inputs (https://detectron2.readthedocs.io/tutorials/models.html#model-input-format).

[zhaoxuyan]

You just need to call model with a batch of inputs (https://detectron2.readthedocs.io/tutorials/models.html#model-input-format).

@ppwwyyxx Thank you very much for your reply~
Can you tell me how to set up a batch of pictures in /demo/demo.py

img = read_image(path, format="BGR")
demo.run_on_image(img)

It can only infer one image at a time, how to infer a batch of images?
Because I have a lot of remaining gpu memory when infer one image at a time, and I want to get the results of multiple video streams(a batch of images) by doing only one inference

[ppwwyyxx]

demo.py does not support batch inference.

[zhaoxuyan]

demo.py does not support batch inference.

So how should I implement it please?

[ppwwyyxx]

https://detectron2.readthedocs.io/tutorials/models.html#use-models explains how to use a model.

[keko950]

You just need to call model with a batch of inputs (https://detectron2.readthedocs.io/tutorials/models.html#model-input-format).

That link doesn't refers in any way to batch inference, would be nice a more detailed answer since the api doesn't go deeper.

[ppwwyyxx]

The link explains that the input to the model is a list[{"image": ...}, ...]. As long as the length of the list > 1, you're doing batch inference.

[keko950]

I was thinking in DefaultPredictor, not model class itself, my bad!
Working like a charm, thank you!

[zhaoxuyan]

The link explains that the input to the model is a list[{"image": ...}, ...]. As long as the length of the list > 1, you're doing batch inference.

@keko950 @ppwwyyxx
Thank you for your reply. Another question:
if my list[{"image": ...}, ...] has 8 images, how to specify multi gpu_num = 4 in DefaultPredictor and each gpu infer with 8/4=2 images?

[ppwwyyxx]

DefaultPredictor uses 1 GPU.

[darkAlert]

I create a predictor as follows:

	cfg = get_cfg()
	cfg.merge_from_file(path_to_cfg)
	cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 
	cfg.MODEL.WEIGHTS = path_to_weights
	predictor = DefaultPredictor(cfg)
        
        img = cv2.imread(img_path)
        img_list = [{"image":img}]
        predictor(img_list)

And then I get the following error:

File "/home/darkalert/builds/pytorch/build/lib.linux-x86_64-3.6/torch/autograd/grad_mode.py", line 49, in decorate_no_grad  return func(*args, **kwargs)
File "/home/darkalert/builds/detectron2/detectron2/engine/defaults.py", line 172, in __call__height, width = original_image.shape[:2] AttributeError: 'list' object has no attribute 'shape'

[ppwwyyxx]

@darkAlert as said above, you need to call a model, not predictor, with batch of inputs.
As the docs (https://detectron2.readthedocs.io/modules/engine.html#detectron2.engine.defaults.DefaultPredictor) said the DefaultPredictor does not support batch of inputs.

[jiangkansg]

I tried to follow the instructions, and this is the code:

cfg = get_cfg()
cfg.merge_from_file("./detectron2_repo/configs/COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7 # set threshold for this model
cfg.MODEL.WEIGHTS = "detectron2://COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x/137849621/model_final_a6e10b.pkl"

from detectron2.modeling import build_model
model = build_model(cfg) # returns a torch.nn.Module

from detectron2.checkpoint import DetectionCheckpointer
DetectionCheckpointer(model).load("detectron2://COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x/137849621/model_final_a6e10b.pkl")

img = cv2.imread("./input.jpg")
img = np.transpose(img,(2,0,1)) # change from HWC to CHW
img_tensor = torch.from_numpy(img)
inputs = [{"image":img_tensor}, {"image":img_tensor}]
outputs = model(inputs)

The error message is:
/content/detectron2_repo/detectron2/modeling/proposal_generator/rpn_outputs.py in _get_ground_truth(self)
260 # Concatenate anchors from all feature maps into a single Boxes per image
261 anchors = [Boxes.cat(anchors_i) for anchors_i in self.anchors]
--> 262 for image_size_i, anchors_i, gt_boxes_i in zip(self.image_sizes, anchors, self.gt_boxes):
263 """
264 image_size_i: (h, w) for the i-th image

TypeError: zip argument #3 must support iteration

Did I miss anything? Thank you.