401 | Invalid Credential - While Accessing GCS Resources | Expiry Delta

[raj-prince]

Hi Team,

I am working on gcsfuse. Recently, get introduced with a very rare issue in gcsfuse while reading some resource from Google Cloud Storage (GCS). The exact error, we get 401: Invalid Credentials.

Workflow in GCSFuse:

• GCSFuse creates TokenSource with ADC, this token source is used to fetch token while making any GCS call. 99.9% it works perfectly fine. But sometime/very rarely it gives error Invalid Credential.
• I tried some experiment to repro this issue, one of them was changing the expiryDelta time to -1 minute. After that we can reproduce this issue more frequently.
• I feel, either there would be some mismatch in clock using which we check either the token is expired or not on the server OR there would be delay in the request which cross the buffer time of 10 seconds (in expiryDelta).

We would like to know -

• Can we introduce any API for the client using that we can change the expiryDelta value?
• How to debug this issue more frequently to fix this? Is there any way to create a token with very less expiry time like (10secs or 20 secs)?

Please let me know, if you need more information related to the issue.

Regards,
Prince Kumar.

[rolandshoemaker]

I don't think we'd want to provide an API to change the global expiryDelta, would a new TokenSource constructor that allows setting the delta work? Something along the lines of (name are arbitrary):

// ReuseTokenSource returns a TokenSource which repeatedly returns the same token as long as it's valid,
// starting with t. When it's cached token is invalid, a new token is obtained from src. The expiration time
// of a token is calculated as t.Expiry.Add(-earlyExpiry), so that tokens can be refreshed with enough time
// before they expire.
func ReuseTokenSourceWithExpiry(t *Token, src TokenSource, earlyExpiry time.Duration) TokenSource

For many users this would likely need to be wired up to FindDefaultCredentialsWithParams by adding a new field to CredentialsParams, etc.

[raj-prince]

Yes, we are looking for similar stuff. Are you planning to support this constructor?

[tmdiep]

+1 for this feature request. There are also issues with refreshing access tokens when using GKE Workload Identity: #634.