Add Exemplar Support to Client Java

[joe-elliott]

Feature Request:
Addition of exemplar support to the previously merged OM support: #615

Thanks for working on this!

[fitzoh]

Any thoughts on API design?

For reference, the client_golang implementation was added in prometheus/client_golang#706 :

//counters
Add(float64)
AddWithExemplar(value float64, exemplar Labels)
//histograms
Observe(float64)
ObserveWithExemplar(value float64, exemplar Labels)

I'm inclined to follow the golang pattern and use distinct method names for exemplar method calls to reduce the risk of people confusing exemplar labels and metric labels.

Does this seem reasonable?

incWithExemplar(double value, String... exemplarLabels)
observeWithExemplar(double value, String... exemplarLabels)

Or would a key/value wrapper be preferred?

incWithExemplar(value, Label.of("key", "value"))

Adding a label class that's only used for exemplars feels a little weird.

[fstab]

In OpenTelemetry for Java, the current span context is always available via

Span.current().getSpanContext()

So the question is, what should our recommended way of dealing with exemplars in Java be? Explicit API calls where the user passes the trace ID as parameter, or implicit integration with OpenTelemetry, i.e. client_java adds exemplars automatically under the hood if a Span context is available?

One argument for the implicit integration is that this will still work if the user does not call the client_java API directly, but uses this library indirectly via Micrometer. In that case, the user will not be able to call incWithExemplar(), but the Java client library will still be able to add examplars by getting them from the current OpenTelemetry span context.

[fitzoh]

One idea that had crossed my mind for exemplar recording would be initializing the counter/histogram with an optional Supplier<Exemplar>, where the supplier might be a lambda that pulls a value from an MCD/trace context or closes over some over system that it can get metadata from. That gets a little tricker if you need to define how often it gets invoked. Do you get an exemplar every time a value is recorded? Every nth invocation? At a fixed rate?

The other option is to completely punt on that question and keep client_java simple with by using something like incWithExemplar and pushing the complexity to micrometer/OT/spring cloud sleuth and let them control when a value is recorded with/without an exemplar.

[fstab]

I am starting to love the idea that users enable OpenTelemetry in their application and automagically client_java will pick this up and provide examplars without code change. That would be an awesome default behavior.

We could implement the Supplier<Exemplar> as a global object shared by all metrics, and register it in the CollectorRegistry. By default it could be using the span context from OpenTelemetry if available. If users want to provide their own implementation they could override this in the CollectorRegistry.

I don't see any harm in providing an exemplar for each metric for each scrape.

[fstab]

FYI I just pushed an otel-exemplars branch with an integration test running a Spring application with the open telemetry java agent attached. It does not do much yet, id just prints the trace ID and greps some metrics, but it could be a good playground to try different approaches.

[fstab]

I just opened a PR #652. Comments are welcome :)

[tcolgate]

General comment...
Exemplars do come at a cost on the prometheus server side, so it might be best to make them opt-in for specific metrics.
Also, a manual/explicit method of adding exemplars, for non-otel users, would be strongly desirable (for me atleast).

[fstab]

The final implementation is now merged to the next-release branch. There are multiple ways to make them opt-in for specific metrics if OpenTelemetry tracing is enabled. And there are methods for explicitly adding exemplars. Thanks for the discussion.