[Instrumentation] Design and implement code instrumentation

[greged93]

Define the various parts of the codebase that would require instrumentation and how we wish to instrument them (metrics-rs, prometheus, grafana, open telemetry).

- [ ] Check if we can change the tracing level dynamically: possible using this layer https://docs.rs/tracing-subscriber/latest/tracing_subscriber/reload/index.html

[greged93]

We wish to monitor the following metrics for the below components of the node to monitor the correct operation of the node:

• Sequencer: gas per second and transaction throughput for the sequencer's full loop (build payload attributes, issue forkchoice update with attributes and issue final forkchoice state).
• Watcher: counter on the L1 messages, batch commits and batch finalizations. counter on the reorgs/gaps. Histogram of L1 reorgs depths.
• Derivation pipeline: counter on derived batch, gauge on block derived per second.
• Indexer: gauge on time to index a notification.
• Signer: gauge on signing duration.

[frisitano]

This looks good as a foundation. In the future, we may want to consider block import latency, network statistics such as the number of messages gossiped and received, L1 message inclusion latency (not relevant for finalized message inclusion). I think we can consider these in more detail in the future and only implement what you have specified, unless you have a strong opinion to include any of these.

[greged93]

I think the block import and maybe the L1 consolidation latencies could be interesting as well, what do you think?

[frisitano]

Yes, I think that sounds good, provided it doesn't add too much overhead. I would propose we land a first version as soon as possible so we can get it plumbed and get initial results quickly. Provided these metrics don't add too much overhead, then let's include them in scope for the first iteration. When doing the implementation, if they raise complications, then we can defer to a later iteration.

[frisitano]

@greged93 can we close this?

[greged93]

I think we can, the dynamic tracing is a huge performance impact, so better to careful set our filters instead.