Seeing maskrcnn_resnet50_fpn + FP32 performance drop by 15% after #5443

[xwang233]

🐛 Describe the bug

Seeing maskrcnn_resnet50_fpn + FP32 performance drop by 15% after #5443

We're running benchmarks for torchvision maskrcnn_resnet50_fpn + FP32 model with pretrained = False, pretrained_backbone = False. After the PR, we saw training throughput drops by ~15% on various GPU models (A100, V100, 3090).

Versions

commit: after #5443

GPU: Nvidia A100, V100, 3090
Library: cuda 11.6 + cudnn 8.3.2

cc @ptrblck @ngimel

[datumbox]

Thanks for reporting.

Prior the #5443, when pretrained=False and pretrained_backbone = False the network's BNs basically didn't work. They were frozen in place to their initial values and the network wouldn't update them (no gradient, no mini-batch statistics updates). After the PR, this bug is fixed, the BN layers do what they are supposed to be doing and thus the perceived performance decrease. Given that BNs account for 1/3 of the total layers of the network, the 15% slowdown is not surprising.