Configuring different custom metrics with spring actuator with different scrape intervals from Prometheus

[debashish-github]

Hi,
I have a set of custom metrics using spring actuator/micrometer framework to spit out metrics compatible with prometheus.
I want some of these metrics to be scraped at 10 second interval and some that are process/thread heavy to be scraped every 2 minutes .
But the way it looks we just have one micrometer registry where all metrics are added and a single url that prometheus is configured to hit .so either I can have all the metrics scraped at 10second interval or every 2 minutes .
Is there a way to segregate these metrics so that I can create different jobs in prometheus with different scrape intervals ?

Thanks
Debashish

[wilkinsona]

I am not a Prometheus expert, but as far as I can tell, its static config only allows a target to be configured with a host and port. I cannot see how you could configure Prometheus to scrape different portions of the same target at different intervals.

Can you share some information about how you would configure Prometheus and the requests that it would make to your Spring Boot application to scrape its metrics? That would allow us to understand what’s needed on the Spring Boot side. It may be possible already with some manual configuration or it may require some enhancements first.

[debashish-github]

So I was thinking of creating a seperate Actuator EndPoint using @WebEndpoint. So for example I have metrics A, B, C, D and E . I want metrics A, B and C to be scraped at 10 second interval. So I create normal custom metric and add them to the micrometer registry . This will have default prometheus end point acutator/prometheus. For metric C and D it create a custom endPoint say actuator/mycustomEndPoint and in its @ReadOperation I create metrics D and E and add them to the @Autowired registry. I tried this and it works great for Counters. It only gets updated when I hit the url actuator/customEndPoint and when I access url actuator/prometheus it returns the value that was updated. The API I use here is registry.counter("custom.health.metric").increment(Math.random()); When I try this for gauge using the API registry.gauge("custom1_azure_storage_connection", isTableConnectionAvailable()); to get the guage value of connection , this value only gets updated at prometheus end point when I first hit the url actuator/myCustomEndPoint . On subsequently hitting this url the actuator/prometheus url doesn't get the updated value . When I use the other API as below: Gauge.builder(MetricConstants.METRIC_NAME_TYPE_CUSTOM + "1", this, value -> isTableConnectionAvailable()) * .description("Azure storage connection check") .tags(Tags.of(Tag.of(MetricConstants.METRIC_TAG_DATE, * sf.format(date)))).baseUnit("azure_storage_connection") .register(registry); Even prometheus scraping calls the isTableConnectionAvailable() since I guess the meter is already registered . So now it gets called from both the end points. I have attached both my files . Working one is CustomHealthEndPoint.java and non working one is AzureConnectionMetricCustom.java.. I have also attached our prometheus.yml configuraiton file. I was planning to add another job with a different endPointUrl having different scrape interval here. Please let me know if you have any futher questions.

[wilkinsona]

Thanks, @debashish-github. Unfortunately, attachments to email replies don’t make it to the issue. Could you please comment directly via the GitHub web UI and attach the files there?

[debashish-github]

Sure please see all the attachments here .

AzureConnectionMetricCustom.txt
CustomHealthEndPoint.txt
prometheus.txt

[debashish-github]

Any updates on this ?