DX-110907: Backport OAuth2 Token Provider support for Azure Workload Identity

[pedroantunesf]

This PR backports HADOOP-18610 from Apache Hadoop branch-3.4 to Dremio's branch-3.3.6.

Original PR: apache#6881
Original Commit: 468b7e5
Author: Anuj Modi (@anujmodi2021)

Adds support for Azure Active Directory (Azure AD) workload identities which integrate with Kubernetes's native capabilities to federate with any external identity provider. This enables ABFS to authenticate using workload identity tokens from files mounted in Kubernetes pods.

Changes Made for branch-3.3.6 Compatibility

1. Merge Conflict Resolutions

AbstractAbfsIntegrationTest.java

• Conflict: Cherry-pick attempted to modify helper methods that don't exist in branch-3.3.6
• Resolution: Added three complete helper methods from branch-3.4:
  • assumeValidTestConfigPresent() - validates test configuration presence
  • assumeValidAuthConfigsPresent() - validates authentication configurations
  • isAppendBlobEnabled() - checks if append blob is enabled
• Impact: +26 lines (these methods already existed in branch-3.4 but are new in branch-3.3.6)

TestAccountConfiguration.java

• Conflict: Method signature and parameter changes in testConfigPropNotFound()
• Resolution:
  • Changed loop variable from CONFIG_KEYS to configKeys parameter
  • Added 4th parameter tokenProviderClassName to setAuthConfig() call
  • Added abfsConf.unset(key); before existing abfsConf.unset(key + "." + accountName);

2. Compilation Fixes for API Differences

AbfsConfiguration.java

• Issue: Missing import for FS_AZURE_ACCOUNT_OAUTH_TOKEN_FILE constant
• Root Cause: branch-3.4 uses wildcard import import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.*; while branch-3.3.6 uses 74 individual explicit imports
• Fix: Added explicit static import:

  import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_ACCOUNT_OAUTH_TOKEN_FILE;

WorkloadIdentityTokenProvider.java

• Issue 1: Invalid @Override annotation on isTokenAboutToExpire() method

• Root Cause: In branch-3.4, AccessTokenProvider has a protected isTokenAboutToExpire() instance method that can be overridden. In branch-3.3.6, this method doesn't exist - only a static method AzureADAuthenticator.isTokenAboutToExpire(token) exists.

• Fix: Removed @Override annotation and changed from super.isTokenAboutToExpire() to AzureADAuthenticator.isTokenAboutToExpire(cachedToken)

• Issue 2: Cannot access parent class's private token field

• Root Cause: The parent AccessTokenProvider class has a private token field that cannot be accessed from child classes

• Fix:

  • Added local cachedToken field: private AzureADToken cachedToken;
  • Modified refreshToken() to cache the token: cachedToken = getTokenUsingJWTAssertion(clientAssertion);
  • Modified isTokenAboutToExpire() to use cachedToken instead of parent's private field

3. Architectural Differences and Limitations

Clock Skew Detection Not Functional:

• In branch-3.4, AccessTokenProvider.getToken() calls isTokenAboutToExpire() polymorphically, allowing WorkloadIdentityTokenProvider to override it with clock skew detection logic
• In branch-3.3.6, AccessTokenProvider.getToken() directly calls AzureADAuthenticator.isTokenAboutToExpire(this.token) (static method), bypassing any child class overrides
• Result: The isTokenAboutToExpire() method in WorkloadIdentityTokenProvider is dead code - it exists for compatibility but is never executed
• Impact: Clock skew detection feature is lost in this backport, but core workload identity authentication functionality works correctly

[jacques-n]

Wow, North of DX-100000. Was the milestone ticket interesting @laurentgo ?