Add a counter for chunks removed from the flush queue but not flushed

[bboreham]

If a chunk is idle and added to the flush queue, then gets new samples shortly before it is dequeued, it will no longer be idle so simply ignored.

It seems like a nice property to be able to add up all reasons why chunks are added to the queue and all reasons they are removed and have them total the same number.

Specifically in the current code, the total cortex_ingester_flush_reasons does not equal the sum of cortex_ingester_chunks_stored_total and cortex_ingester_dropped_chunks_total.

(and saying that makes me realise stored and dropped should be labels on one metric)

[pstibrany]

cortex_ingester_flush_reasons counts series, while cortex_ingester_chunks_stored_total and cortex_ingester_dropped_chunks_total count chunks, so they cannot be compared. We could add new metric or modify cortex_ingester_flush_reasons to cortex_ingester_chunks_flush_reasons.

We can fix incorrect cortex_ingester_flush_reasons for previously-idle-but-not-anymore chunks by moving that increment from where it is now (just after series has been enqueued) to flushUserSeries method, after dequeing and checking flush reason again. WDYT? (I can put that into #2778 PR, if you agree). (Sent separate PR)

[bboreham]

I don't agree this should be fixed simply by not recording why series are added to the queue.

Good point that some metrics count chunks and some count series.