Add Prometheus metrics about cache

[tobikris]

NetBox version

v3.5.7

Feature type

Change to existing functionality

Proposed functionality

Currently the prometheus metrics do not include metrics about the cache. The documentation suggests otherwise

netbox/docs/integrations/prometheus-metrics.md

Line 18 in 0b10131

- Cache hit, miss, and invalidation counters

I could imagine that the change of the cache backend removed this as a side-effect.
As those metrics are pretty useful, I propose to add them back.

Use case

Without the metrics it is pretty hard to determine the effectiveness of the caching. I would like to see if the cache is working as expected or if the general performance is worse than it should be.

Database changes

No response

External dependencies

No response

[kkthxbye-code]

What caching though, there's basically none in use. Specifically what do you want to monitor?

[jsenecal]

What caching though, there's basically none in use. Specifically what do you want to monitor?

Cache hit, miss, and invalidation counters

These guys: Diff

[kkthxbye-code]

@jsenecal - We don't really cache anything, which is why I asked.

[jsenecal]

@kkthxbye-code yeah, I get that, I was replying to the "what". Perhaps the question to @tobikris is more "Why" :)

[kkthxbye-code]

I just want to know what specific objects he needs the cache hit, miss and invalidation counters for, because we cache the release check, config revisions and the RSS widget and that's pretty much it. Nothing were metrics would actually be of use imo.

I'm assuming the why is because he thinks we still cache stuff, so that's not really interesting.

[tobikris]

Your assumption is correct - based on the docs I was expecting to see cache hits etc for modelled objects.
We are having some issues with long response times and wanted to dig into different aspects. This is why we tried to use the Prometheus metrics and realized the cache metrics mentioned in docs were missing.

Thanks for the explanation. I guess this feature request can be closed again as you are right - those metrics are not important if only minor things are cached anyway.

[kkthxbye-code]

@tobikris - Long response times are usually because of high pagination size. Page load time scales pretty linearly with pagination size, so if someone set it to 1000, which is the default max, page load time can be multiple seconds.

Other stuff that slows down load times include custom link columns, custom field of the object type and for prefixes specifically, the utilization column.

[tobikris]

Thanks again. We are aware of the page size and its impact on response times. However, we are a little bit forced to load all elements in one request. This is because the current pagination implementation without cursors does not ensure correctness in case of parallel changes.

But even in our tests using pagination with different page sizes it took about 40 seconds in total to retrieve all 7000 IP addresses. Sweetpoint was at about 250 items per request.
Does that sound about right/expected? Please note that we are currently not running the latest version (v3.2.8) for unrelated reasons.

[jsenecal]

It obviously depends on how fast your CPU cores are, but yeah 7000 IPs is a lot to retrieve. Maybe there is a better way, but a discussion would be more suited for this.