Perf regression caused by #5474

[msullivan]

It looks likes #5474 caused a perf regression to the tune of 5% on a self check and in our internal repos.

@ilevkivskyi said he had an idea for fixing