Create a experimental HealthCheck GRPC Handler

[alanprot]

What this PR does:
WIP/Proposal

Create a experimental HealthCheck GRPC Handler.

This health check will evaluate the targets and immediately terminate the request if a target is marked as unhealthy.

Immediately terminating the request when the target is unresponsive provides two major benefits:

• Prevent overload on distributors
• Avoid trying to call unhealthy SG and so, retry the request on a healthy one faster.

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yeya24]

Can you help me understand the difference between this health check and the existing gRPC client pool https://github.com/cortexproject/cortex/blob/master/pkg/ring/client/pool.go#L60.

Can we just add the health check to the existing pool client? They seem similar functionality to me.

[alanprot]

Can you help me understand the difference between this health check and the existing gRPC client pool https://github.com/cortexproject/cortex/blob/master/pkg/ring/client/pool.go#L60.

Can we just add the health check to the existing pool client? They seem similar functionality to me.

The pool only removes the client when the healthcheck fails, but create a new one right away when a new request come in.

cortex/pkg/ring/client/pool.go

Line 94 in 6a7f44c

func (p *Pool) GetClientFor(addr string) (PoolClient, error) {

This change here is "short circuiting" the requests if the healthcheck fails.

An alternative for this is to create a classic circuit breaker (ex: https://failsafe-go.dev/circuit-breaker/) but i think as a first step using a normal health check config would be simpler.

[yeya24]

LGTM