Optimise HTTP API /queues endpoint

[kjnilsson]

Listing queues with the HTTP API when there are many (1000s) of
quorum queues could be excessively slow compared to the same scenario
with classic queues.

This optimises various aspects of HTTP API queue listings.
For QQs it removes the expensive cluster wide rpcs used to get the
"online" status of each quorum queue. This was previously done before
paging and thus would perform a cluster-wide query for each quorum queue in
the vhost/system. This accounted for most of the slowness compared to
classic queues.

Secondly the query to separate the running from the down queues
consisted of two separate queries that later were combined when a single
query would have sufficed.

This commit also includes a variety of other improvements and minor
fixes discovered during testing and optimisation.

MINOR BREAKING CHANGE: quorum queues would previously only display one
of two states: running or down. Now there is a new state called minority
which is emitted when the queue has at least one member running but
cannot commit entries due to lack of quorum.

Also the quorum queue may transiently enter the down state when a node
goes down and before its elected a new leader.

[kjnilsson]

5k quorum queues on a 3 node dev box.

On this branch:

karl@crake:~$ time curl 'http://crake:15672/api/queues?page=25&page_size=100&name=&use_regex=false&pagination=true' -H 'authorization: Basic Z3Vlc3Q6Z3Vlc3Q=' &>/dev/null

real	0m0.080s
user	0m0.003s
sys	0m0.007s

On main:

karl@crake:~$ time curl 'http://crake:15672/api/queues?page=25&page_size=100&name=&use_regex=false&pagination=true' -H 'authorization: Basic Z3Vlc3Q6Z3Vlc3Q=' &>/dev/null

real	0m1.231s
user	0m0.000s
sys	0m0.006s

I expect on a system with a real network and some load to benefit even more.

[tubemeister]

Thanks, that should make a big difference. A 10s timeout is workable, a 1m timeout isn't.

Any chance of this getting a backport to 3.12?

(PS: I've been locked out of my own bugreport. I've been getting more responses there from both of you but can't respond.)

[michaelklishin]

There is no consensus on whether we want to backport this. #9892 is ready for review but we make no promises.

[michaelklishin]

It was decided that this should be a 3.13-specific change.

[tubemeister]

Yeah I suspected as much from the "breaking change" aspect, that makes sense.

Thanks anyway, something to look forward to in 3.13. ;-)

[tubemeister]

Can confirm, the long timeout when a node is offline is now fixed in 3.13.0

[tubemeister]

Might be useful to note this as fixed this in #9522 as well. I'd do it myself but I've been locked out of my own bugreport. ;-)

[mkuratczyk]

done