server: metrics endpoint

[phymbert]

Issues

• Server /metrics endpoint share the same task event as /health: TASK_TYPE_METRICS. It means metrics are reset on both calls.
• the Process-Start-Time-Unix http response header is not set.
• metrics llamacpp:prompt_tokens_seconds and llamacpp:predicted_tokens_seconds are per slots, while the server actually process llamacpp:prompt_tokens_seconds * n_slots

Proposal

• Add a data params in TASK_TYPE_METRICS to reset the metric bucket only in /metrics
• Add llamacpp:prompt_tokens_seconds_total and llamacpp:predicted_tokens_seconds_total

[ngxson]

I have one idea to improve /health but I'm not sure if it's currently possible or not:

If user only want to call /health to check if the server is busy doing something or not, then this call should not be blocking. Why - because if the call is blocked, then the moment it finishes, it always says that the server is free - meaning health will never be "busy"

The application of this such case is where include_slots is not set, that mean user just want to see how many idle / processing slot. This check can be done easily with a mutex I think. Only when include_slots is set, then we push the request to queue.

[phymbert]

it blocks only during prompt/image processing, llama_decode. It works as expected while slots are waiting for the next token. So, as @ggerganov explained earlier in #5851, we are fine with it. I will just fix the bucket window reset on /health instead of /metrics, add a few more total metrics, and fix json cast to int instead of float on kv cache use ratio.