Refactor: Consider a shared nothing architecture, to reduce thread complexity #200
Description
ATM there's quite a bit of work in managing SHARED state between threads/sub systems. Although this is working well, it's complex. A shared nothing architecture would mean there are no sync issues, however work would need to be done to implement async request response, or maybe just pushing messages between systems. This might actually not be that hard to implement. It's basically just the Actor Mailbox pattern. We already use mail boxes really, but they're not used in clear boundaries between acts and state systems. New mail boxes would replace the existing concurrent collections that we already use and rely on, but in a more structured and clearly delineated fashion.
There are three thread actors atm - the controller, the poller, and the workers. Main sync issues occur because the poller and the controller share access to the WorkManager, PartitionMonitor and ShardManager. This would change it so that only the controller has access to these, and the poller instead pushes messages to it's inbox - partition assignment changes and new polled messages.
Poller would try to make sure controllers Records inbox always has enough queued. And it would push assignment messages to it, and it would be ok that the controller process these messages only at the start of it's cycle. As it would simply drop results from records that aren't assigned anymore, and async remove from inbound queues stale records before they're processed.
The ultimate simplification would be to eliminate the seperate poller thread. I think the main reason there's a seperate thread is because of blocking polls to the broker when there's no data in the partitions. It will wait up until some max. However, two options:
a) go with the interrupting model and interrupt a blocking poll if work arrives for the controller to do.
b) don't block when polling?