Client-side transport hooks

[joybestourous]

Is your feature request related to a problem?

We're trying to add some stats on the number of connections opened, closed, and active on the client side for grpc-java. On the server side, ServerTransportFilter acts as a reasonable hook for this, but there doesn't seem to be anything comparable on the client side that I can find (if there is, please direct me to it!).

Describe the solution you'd like

A hook into the underlying transport information would be really helpful here. In particular I've thought up two options

1. A ClientTransportFilter. I put together an implementation for this that's more or less parallel to the Server side filter. It's a very simple change
2. A Java equivalent of grpc-go's StatsHandler. I think this would be a great generalized solution to metrics in grpc-java. Right now, setting up grpc metrics requires a combination of StreamTracers and Interceptors, but with this strategy folks can instrument their metrics in one localized place and add it via an addStatsHandler API. I do not have a POC for this option, but I can put one together if it's your preference.

Please let me know if either of these solutions are suitable to you folks!

[ejona86]

We're looking at adding these types of metrics for OTel, so something like this might be coming (server-side would clearly be high priority).

Although it might be through a generic "metric plugin API" where gRPC is in charge of the metrics and just calls out to the plugin to record them. (Think: OTel MeterProvider API behind a façade.) The Tracer/callback approach will probably be reserved more for cases that have performance concerns. I think it is still too early to tell what approach we'd do on client-side here.

CC @DNVindhya

[joybestourous]

A big part of the reason that we're instrumenting this ourselves is because our customers can't wait for these metrics, so we'd like a more immediate solution (we did also file an issue with grpc-core: grpc/grpc#34886 for otel connection metrics, but we don't expect the A66 project in general to be complete soon enough for our needs). Based on your response, I see a couple of options:

1. We could contribute to the metric plugin API you mentioned and perhaps push up the timeline for its creation
2. If you folks are still unsure if that API will be the approach you'll take, we could perhaps explore the ClientTransportFilter option in the meantime?

What do you think?

[ejona86]

We have been having meetings for a long time banging out the metric plugin API. It is in a reasonably agreed upon state now, but the caller into this API makes certain decisions that mean we'd need agreement on the OTel metrics.

I see in ClientTransportFilter you expose transportShutdown. That isn't part of ServerTransportFilter. Is that not necessary on server-side?

The branch cut is next week, so that may push it a bit to get ClientTransportFilter in a semi-stable API. If we break the API (e.g., rename a method, but all functionality is still available), does that cause you much harm?

[joybestourous]

We have been having meetings for a long time banging out the metric plugin API. It is in a reasonably agreed upon state now, but the caller into this API makes certain decisions that mean we'd need agreement on the OTel metrics.

Makes sense, we're looking forward to seeing this happen!

I see in ClientTransportFilter you expose transportShutdown. That isn't part of ServerTransportFilter. Is that not necessary on server-side?

I'm not sure why it isn't exposed on the server side. On the client side I figured it's useful because it exposes the status/ shutdown reason, but I suppose we don't quite need that.

The branch cut is next week, so that may push it a bit to get ClientTransportFilter in a semi-stable API. If we break the API (e.g., rename a method, but all functionality is still available), does that cause you much harm?

Breaking the API shouldn't cause us much harm at all so long as functionality is still there. Appreciate you taking a look at it, I'll address your comments this afternoon.