Make discovery almost free and continuous after all is discovered

[Gerold103]

Router discovery is a heavy operation - it downloads all the buckets from all the storages. Sometimes that creates problems on the storages, when they need to send thousands of buckets.
After #210 the situation should become less problematic. But still not perfect. For the ideal discovery a router should be able to stop downloading anything from storages, when all is discovered. However still send requests to the storages to check if anything changed since the last discovery.
Need to introduce some kind of 'cookie'. A value, which a router would save when all is discovered. When it is sent to storage, it either returns nothing if the cookie is the most actual, or returns buckets if anything changed since the last discovery.
Cookie probably could be a pair {bucket_generation, bucket_timestamp}. Bucket generation is a volatile counter incremented on every _bucket change by every storage. Only generation is not enough, because the storage may change _bucket, restart, and the counter can become exactly the same as it was before restart, but with different bucket set.
Timestamps should protect from that. Maybe just timestamp would be enough though.

[Gerold103]

There is a new idea from @unera. It is about storing a logical clock value with each bucket. Like LSN. Each replicaset has a monotonically growing sequence of numbers called BSN - bucket sequence number. Each update of any bucket in _bucket causes the storage BSN increment and takes the new BSN value.

There is an index built over BSN field in _bucket space. When router connects to a storage, it assumes the last known BSN is 0. And downloads all the buckets having BSN >= 0. After it is done, it remembers the latest downloaded BSN locally.

On each next bucket discovery request to that storage it will request buckets having BSN > last known. If there aren't any, it means nothing changed.

If there are any updates, the router will download only them, thanks to the index over BSN values. Then it remembers the last BSN again, and the process is repeated forever.

That looks very good. But has 2 issues which I could spot:

Where to store the latest BSN when buckets are deleted?

If all storage buckets are deleted (weight = 0), then there is no place to store the latest BSN of the storage.

• One option is to store it in _schema somewhere.
• Another option - store a dummy bucket in _bucket with ID 0 and the max BSN.

If it is ok for routers to redownload _bucket when reconnect to a storage happens, then:

• Another option is to store the BSN in memory, not in a space. But on restart initialize it as max(box.info.lsn, _bucket.index.bsn:max()).
• Another option is to store BSN as the current timestamp in nanoseconds + some volatile in-memory number to avoid collisions when several buckets are changed in a single event loop iteration.

If the routers must be able not to redownload _bucket even if connection breaks or the storage restarts, then the BSN must be persisted somewhere.

Can't download removal of sent and deleted buckets

If a bucket is deleted completely, the storage BSN is updated, but it is not possible for the router to learn which buckets were deleted. Which makes the feature hardly usable because during rebalancing most of the storages will delete their buckets, and only a few storages will receive them.

• One option is to never delete buckets from _bucket. Even if a bucket was sent, and its data was deleted - keep it in SENT state there to preserve the generation. A positive side effect would be that we can track bucket movements over the cluster any time. Negative effect - with millions of buckets it will take megabytes of metadata on each replicaset to store them all.

If it is ok for the routers to redownload the buckets on reconnect/restart, then:

• Another option - do like above, but on restart remember the current bucket BSN and cleanup the SENT/GARBAGE buckets with BSN <= the remembered one. It is guaranteed no router will need to download their deletion, because all the connections were closed on restart.
• Next option - do like in the first idea, but also make routers track on the storage in their box.session which BSN did they download. And allow to delete only those buckets which are behind any router's session BSN.

I think that if the feature is worth doing, then it should allow not to redownload entire _bucket from each replicaset even on restart.

[Gerold103]

The LSN idea might help to solve this issue too: #214.