Add metrics for expirationd stats

[no1seman]

It wiil be great to add expirationd stats metrics to have an availability to see them on timeline.

[filonenko-mikhail]

It's not our business, sorry. Gently kindly closing.

[opomuc]

why is this not our business?

it can be used by our customers on various occasions.

[filonenko-mikhail]

it's expirationd business

[opomuc]

I think we should not forget dashboard for such metrics. what do you think @DifferentialOrange ? can we somehow make custom panels that are displayed only if certain metrics are in the result?

[DifferentialOrange]

I think we should not forget dashboard for such metrics. what do you think @DifferentialOrange ? can we somehow make custom panels that are displayed only if certain metrics are in the result?

AFAIK, Grafana do not support this (and I doubt it ever will). But if we do tarantool/grafana-dashboard#128 , "excessive" panels will be less annoying.

[no1seman]

Seems that professional likers needed: grafana/grafana#13468

[vrogach2020]

Suggested RFC:

1. Add export method for metric that returns stats of all active tasks in formatted form for metrics.
   For example get_all_tasks_stats():

task1:
    checked_count = 1000
    expired_count = 122
    restarts = 3
    working_time = 600

task2:
    checked_count = 2000
    expired_count = 12
    restarts = 0
    working_time = 30

Stats is already collected by expirationd so this data can be obtained via expirationd.stats method.

2. Add a metric registration method to enable expiration stats publication.

This method should register a callback in metrics.
As a target metric type we can use gauge that represent a snapshot of the collected stats.

For example: expirationd.enable_metrics().

local metrics = require('metrics')

local expiration_gauge = metrics.gauge('expirationd', 'Expirationd statistics')

metrics.register_callback(function()
    local stats = json.encode( get_all_tasks_stats() )
    expiration_gauge:set(stats, {app = 'tarantool'})
end)

3. This metric must support prometheus.