Use frozen BN only if pre-trained backbone

[datumbox]

Currently, the majority of our Detection models replace the BatchNorm2d layers with FrozenBatchNorm2d. This is a reasonable mitigation that improves the stability of training for small batch-sizes. Unfortunately, our current implementation freezes the BNs even when they are completely randomly initialized. Since FrozenBatchNorm2d freezes both the running stats and the affine parameters, its parameters get initialized and fixed to values to:

vision/torchvision/ops/misc.py

Lines 30 to 33 in 0c2373d

	self.register_buffer("weight", torch.ones(num_features))
	self.register_buffer("bias", torch.zeros(num_features))
	self.register_buffer("running_mean", torch.zeros(num_features))
	self.register_buffer("running_var", torch.ones(num_features))

Consequently, the BN layers are effectively completely disabled for those who try to train the models from scratch.

This PR fixes the issue by replacing the BNs with FrozenBNs when at least some pre-trained weights are loaded.

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit d222c46 (more details on the Dr. CI page):

---

• 1/1 failures introduced in this PR

---

1 failure not recognized by patterns:

Job	Step	Action
cmake_macos_cpu	curl -o conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
sh conda.sh -b
source $HOME/miniconda3/bin/activate
conda install -yq conda-build cmake
packaging/build_cmake.sh
	🔁 rerun

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[fmassa]

Thanks!

[fmassa]

nit: if the model was trained for detection with large batch sizes from scratch, and then we finetune it afterwards (still with large batch sizes) then in this case we would be using FrozenBatchNorm.

This is an ok heuristic, but hints that we might want to make this an explicit parameter from the constructor in the future