GRPC will keep retrying (until RPC timeout) if a user uses a bad service account key

[dzou]

What version of gRPC-Java are you using?

1.27.2

What is your environment?

Linux, JDK8

What do you see?

When a user uses an invalid service account key (like one that was deleted), it is treated as an UNAVAILABLE error. Unavailable errors are interpreted as ones that should be retried by client libraries; consequently the application will attempt to retry this operation with invalid credentials until the RPC timeout is reached (usually 10 minutes).

What do you expect to see instead?

When a user uses an invalid service account key to authenticate with GRPC, it should yield an UNAUTHENTICATED error which indicates the operation should not be retried.

Steps to reproduce the bug

This is most easily reproduced through experimenting with the client libraries.

1. Create a service account and service account key, download it.
2. Delete the key entry in Cloud Console (so the downloaded key is no longer valid).
3. Authenticate with some google service using that key. Here is an example using Pub/Sub like this.

Additional details:

• Many users have reported being confused by this behavior since 2018: Bad credentials file causes Pub/Sub topic creation to hang googleapis/google-cloud-java#3573

Suggested fix:

The code that controls this behavior is in GoogleAuthLibraryCallCredentials. I don't think all IOExceptions should be retried though.

In this case, one can see that the exception has a .getCause() which is HttpResponseException and it has .getStatusCode() == 400 which indicates a bad request. This is the error thrown if the user provides an invalid service account key.

Would it be possible to modify it so that if it is an IOException, it will examine the getCause() of the exception and throw UNAUTHENTICATED if the cause is HttpResponseException with status code 400?

Example of exception that you see:

th [] threw exception [Request processing failed; nested exception is com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata] with root cause

com.google.api.client.http.HttpResponseException: 400 Bad Request
{"error":"invalid_grant","error_description":"Invalid JWT Signature."}
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113) ~[google-http-client-1.34.2.jar:1.34.2]
	at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:441) ~[google-auth-library-oauth2-http-0.20.0.jar:na]
	at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:157) ~[google-auth-library-oauth2-http-0.20.0.jar:na]
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:145) ~[google-auth-library-oauth2-http-0.20.0.jar:na]
	at com.google.auth.oauth2.ServiceAccountCredentials.getRequestMetadata(ServiceAccountCredentials.java:603) ~[google-auth-library-oauth2-http-0.20.0.jar:na]
	at com.google.auth.Credentials.blockingGetToCallback(Credentials.java:112) ~[google-auth-library-credentials-0.20.0.jar:na]
	at com.google.auth.Credentials$1.run(Credentials.java:98) ~[google-auth-library-credentials-0.20.0.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_181-google-v7]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181-google-v7]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_181-google-v7]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:295) ~[na:1.8.0_181-google-v7]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181-google-v7]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_181-google-v7]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181-google-v7]

[sanjaypujare]

@dzou Based on getCause() being 400/Bad Request I think it is a stretch to infer UNAUTHENTICATED as the root cause. However the JSON snippet in the status text does indicate an authentication issue and that could be potentially used to infer UNAUTHENTICATED if that is what we get in all such cases.

[dzou]

I see, then forget about UNAUTHENTICATED; However I think 400/Bad Request should be an indication that the request should not be retried, no?

[sanjaypujare]

That's debatable. Without more information I can see a "bad request" succeeding after retrying (some kind of a race condition or simply 400/Bad Request being the incorrect code returned from the server). So one could argue that it is safe to retry a bad request.

[dzou]

Ah, so I am interpretting the 400 error-codes as an error being made on the caller's side - see this for what I had in mind: https://stackoverflow.com/questions/47680711/which-http-errors-should-never-trigger-an-automatic-retry

I was thinking basically that a 400-code would mean the user's request is incorrect (i.e. in this case making a request with a service account key that does not exist), and if we know the user's request is invalid then we need not retry it.

[sanjaypujare]

Consider gRPC's retry feature as part of an advanced framework wrapping an existing protocol like HTTP (HTTP2 to be more precise). Some users would expect the framework to retry even in cases of 400/Bad Request just because the error code sounds like a catch-all generic error code. But I do see where you are coming from and if many others feel the same way I guess the change you want can be made. But it will have to be more than just mapping 400/Bad Request to UNAUTHENTICATED since UNAUTHENTICATED doesn't sound right.

[sanjaypujare]

I agree that logic of onFailure inside GoogleAuthLibraryCallCredentials can be tweaked as follows: in case of IOException check if it is an com.google.api.client.http.HttpResponseException and if the status code says 401 (UNAUTHORIZED) then call applier.fail with Status.UNAUTHENTICATED similar to the current else clause. However that still requires modification in the google-auth-library to generate 401 instead of 400 in this case. Once that is done we can reopen this issue and someone can open a PR for GoogleAuthLibraryCallCredentials.

[ejona86]

@jiangtaoli2016, how do you think this should be fixed? It probably impacts all languages.

I think there are two things to discuss here

1. Normally we auto-promote ServiceAccountCreds to ServiceAccountJwtAccessCredentials, which avoids exchanging a JWT for an OAuth token (via plain HTTP). It appears your system was not doing this exchange, probably because you specified scopes. If grpc-java ignored the scopes and auto-promoted in this case as well, it would have avoided your issue because the server would have responded with UNAUTHENTICATED. C core made this change earlier, but it was not made to grpc-java

2. The status code to use when there is a failure getting an OAuth token. That used to be UNAUTHORIZED but was changed to UNAVAILABLE because the OAuth exchange can fail due to normal I/O problems. The HttpResponseException exception is very awkward for gRPC to dig into, as it is part of google-http-java-client which is an implementation detail of google-auth-library-java. It is unclear how this would be resolved; it would probably need enhancements to the google-auth-library-java API

A hacky "fix" for (2) is easy, but I don't think it puts us on solid ground and would just bite us in the future as an implementation change in google-auth-library-java could change the status code. And we'd need to figure out an equivalent solution in each language. Spending our time on (1) seems better, and while it does side-step the issue it also provides other benefits.

[jiangtaoli2016]

The token change from JWT server account creds to OAuth token is via
https://accounts.google.com/o/oauth2/token.

@ejona86 Could you point the grpc core change that you refer? I think it makes sense to have all languages behave consistently.

[ejona86]

@jiangtaoli2016, no, I can't point to the grpc core change. It was a long time ago and just mentioned to me "in person." I can't even point to the current behavior in c core.