Contract reaper thread not running

[citrus-it]

#5318 caused sled-agent to run out of contracts after 216 failed attempts to boot the switch zone.
There is a contract reaper thread that releases contracts after running commands in the context of another zone, but that was for some reason not running.

BRM44220001 # pstack `pgrep sled-agent` | demangle | grep contract_reaper
BRM44220001 #

Compare this with a running sled in dogfood:

BRM44220011 # pstack `pgrep sled-agent` | demangle | grep contract_reaper
 000000000261ecec illumos_utils::running_zone::zenter::contract_reaper::hca301d33703c3725 () + 19c

I haven't looked at the code yet, but we are presumably starting the contract reaper thread too late, it should be created before we ever run a command in a non-global zone context or we will leak contracts.

[jgallagher]

we are presumably starting the contract reaper thread too late

This looks right to me. We start the contract reaper thread in SledAgent::new(), but we don't land in SledAgent::new() until we've gotten a StartSledAgentRequest from RSS (or from our local disk if we're cold booting after RSS).

We should probably start it somewhere earlier in the startup process - maybe in BootstrapServer::start() or even in main before we start the bootstrap server?

Looking back at the original PR, we did talk about this (#3839 (comment)); it looks like there was some logistical problem getting the Logger earlier; we'll probably need to address that to fix this.