HADOOP-19406. ABFS: [FNSOverBlob] Support User Delegation SAS for FNS Blob #7523
+228
−43
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Test Results============================================================
|
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
| @@ -250,6 +251,7 @@ public void initialize(URI uri, Configuration configuration) | |||
| try { | |||
| if (abfsConfiguration.getAuthType(abfsConfiguration.getAccountName()) == AuthType.SAS && // Auth type is SAS | |||
| !tryGetIsNamespaceEnabled(new TracingContext(initFSTracingContext)) && // Account is FNS | |||
| abfsConfiguration.getFsConfiguredServiceType() == DFS && // Service type is DFS | |||
There was a problem hiding this comment.
Use the constant for DFS
There was a problem hiding this comment.
It is using the AbfsServiceType.DFS constant here
| @@ -75,11 +77,16 @@ public String getDelegationSAS(String accountName, String containerName, String | |||
| case SASTokenProvider.GET_STATUS_OPERATION: | |||
| sp = "e"; | |||
| break; | |||
| case SASTokenProvider.LIST_OPERATION_BLOB: | |||
| sp = "l"; | |||
| sr="c"; | |||
| final URL url = createRequestUrl(destination, | ||
| abfsUriQueryBuilder.toString()); | ||
| final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders(); | ||
| final AbfsRestOperation successOp = getSuccessOp( | ||
| AbfsRestOperationType.RenamePath, HTTP_METHOD_PUT, | ||
| url, requestHeaders); | ||
| successOp.setMask(); |
There was a problem hiding this comment.
Why is this performed only in the rename case ? By default we mask for all cases if auth type is SAS
There was a problem hiding this comment.
My bad, yes we mask for all SAS cases
| @@ -91,14 +91,17 @@ public ITestAzureBlobFileSystemDelegationSAS() throws Exception { | |||
| public void setup() throws Exception { | |||
| isHNSEnabled = this.getConfiguration().getBoolean( | |||
| TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT, false); | |||
| Assume.assumeTrue(isHNSEnabled); | |||
| if(!isHNSEnabled){ | |||
There was a problem hiding this comment.
Add more tests around user delegation SAS specific to FNS Blob and see it works correctly for all cases of implicit, explicit, root path, container path
There was a problem hiding this comment.
Added some new tests. Please review
| @@ -663,13 +663,13 @@ To know more about how SAS Authentication works refer to | |||
| [Grant limited access to Azure Storage resources using shared access signatures (SAS)](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview) | |||
|
|
|||
| There are three types of SAS supported by Azure Storage: | |||
| - [User Delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas): Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts. It is Identity based SAS that works at blob/directory level) | |||
| - [User Delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas): Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts or HNS-Disabled Blob Storage accounts. It is Identity based SAS that works at blob/directory level) | |||
There was a problem hiding this comment.
This recommended line should say HNS only
There was a problem hiding this comment.
Yes recommended only for HNS-DFS.
Supported on HNS-DFS and FNS-BLOB
Not supported on FNS-DFS
This should be clearly conveyed
There was a problem hiding this comment.
Makes sense, rectified
| - [Service SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-service-sas): Global and works at container level. | ||
| - [Account SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-account-sas): Global and works at account level. | ||
|
|
||
| #### Known Issues With SAS | ||
| - SAS Based Authentication works only with HNS Enabled ADLS Gen2 Accounts which | ||
| is a recommended account type to be used with ABFS. | ||
| - SAS Based Authentication works with HNS Enabled ADLS Gen2 Accounts (which |
There was a problem hiding this comment.
Does SAS work correctly with FNS DFS ? We should say FNS here only if we confirm the behaviour for both the endpoints
| @@ -737,7 +737,7 @@ the following configurations apart from above two: | |||
|
|
|||
| - **Security**: More secure than Shared Key and allows granting limited access | |||
| to data without exposing the access key. Recommended to be used only with HNS Enabled, | |||
| ADLS Gen 2 storage accounts. | |||
| ADLS Gen 2 storage accounts or HNS-Disabled Blob Storage accounts. | |||
There was a problem hiding this comment.
again remove non HNS for recommended
anujmodi2021
requested changes
Apr 2, 2025
...-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java
Show resolved
Hide resolved
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
Show resolved
Hide resolved
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsErrors.java
Outdated
Show resolved
Hide resolved
| @@ -663,13 +663,13 @@ To know more about how SAS Authentication works refer to | |||
| [Grant limited access to Azure Storage resources using shared access signatures (SAS)](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview) | |||
|
|
|||
| There are three types of SAS supported by Azure Storage: | |||
| - [User Delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas): Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts. It is Identity based SAS that works at blob/directory level) | |||
| - [User Delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas): Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts or HNS-Disabled Blob Storage accounts. It is Identity based SAS that works at blob/directory level) | |||
There was a problem hiding this comment.
Yes recommended only for HNS-DFS.
Supported on HNS-DFS and FNS-BLOB
Not supported on FNS-DFS
This should be clearly conveyed
...azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelegationSAS.java
Show resolved
Hide resolved
manika137
force-pushed
the
HADOOP-19406_UDSSupport
branch
3 times, most recently
from
0fd1ded to
1676667
Compare
manika137
force-pushed
the
HADOOP-19406_UDSSupport
branch
from
1676667 to
a01beb7
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
💔 -1 overall
This message was automatically generated. |
anujmodi2021
approved these changes
Apr 14, 2025
| - [User Delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas): Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts. It is Identity based SAS that works at blob/directory level) | ||
| - [User Delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas): | ||
| Recommended for use with ABFS Driver with HNS Enabled ADLS Gen2 accounts. It | ||
| is Identity based SAS that works at blob/directory level) | ||
| - [Service SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-service-sas): Global and works at container level. | ||
| - [Account SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-account-sas): Global and works at account level. | ||
|
|
||
| #### Known Issues With SAS |
There was a problem hiding this comment.
SAS-based authentication works with HNS-enabled ADLS Gen2 accounts (recommended for use with ABFS) and is also supported with non-HNS (FNS) Blob accounts. However, it is not supported with FNS-DFS accounts. - better to frame like this
| @@ -68,18 +70,25 @@ public String getDelegationSAS(String accountName, String containerName, String | |||
| case SASTokenProvider.DELETE_RECURSIVE_OPERATION: | |||
| sp = "d"; | |||
| sr = "d"; | |||
| sdd = Integer.toString(StringUtils.countMatches(path, "/")); | |||
| sdd = path.equals("/")? "0": Integer.toString(StringUtils.countMatches(path, "/")); | |||
There was a problem hiding this comment.
Converted it to path.equals(ROOT_PATH)? "0": Integer.toString(StringUtils.countMatches(path, "/"));
Did not change the last "/" part for readability purpose
| case SASTokenProvider.LIST_OPERATION: | ||
| sp = "l"; | ||
| sr = "d"; | ||
| sdd = path.equals("/")? "0": Integer.toString(StringUtils.countMatches(path, "/")); |
| @@ -1056,7 +1058,7 @@ public AbfsRestOperation flush(byte[] buffer, | |||
| final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder(); | |||
| abfsUriQueryBuilder.addQuery(QUERY_PARAM_COMP, BLOCKLIST); | |||
| abfsUriQueryBuilder.addQuery(QUERY_PARAM_CLOSE, String.valueOf(isClose)); | |||
| String sasTokenForReuse = appendSASTokenToQuery(path, SASTokenProvider.FIXED_SAS_STORE_OPERATION, | |||
| String sasTokenForReuse = appendSASTokenToQuery(path, SASTokenProvider.WRITE_OPERATION, | |||
There was a problem hiding this comment.
Is using WRITE_OPERATION for the flush call a good idea? Should we create another operation type for this?
There was a problem hiding this comment.
We have the same operation type for DFS flush call as well. We can keep it as it is I think
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
anmolanmol1234
approved these changes
Apr 23, 2025
|
💔 -1 overall
This message was automatically generated. |
manika137
force-pushed
the
HADOOP-19406_UDSSupport
branch
from
d8a2816 to
82ef314
Compare
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
manika137
added a commit
to ABFSDriver/AbfsHadoop
that referenced
this pull request
Apr 24, 2025
… Blob (apache#7523) Contributed by Manika Joshi Reviewed by Anmol Asrani, Manish Bhatt, Anuj Modi Signed off by Anuj Modi<[email protected]>
anujmodi2021
pushed a commit
that referenced
this pull request
Apr 25, 2025
… Blob (#7523) (#7649) Contributed by Manika Joshi Reviewed by Anmol Asrani, Manish Bhatt, Anuj Modi Signed off by Anuj Modi<[email protected]>
zhtttylz
pushed a commit
to zhtttylz/hadoop
that referenced
this pull request
Apr 27, 2025
… Blob (apache#7523) Contributed by Manika Joshi Reviewed by Anmol Asrani, Manish Bhatt, Anuj Modi Signed off by Anuj Modi<[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
JIRA: https://issues.apache.org/jira/browse/HADOOP-19406
In ABFS Driver, user delegation SAS is currently only supported by hierarchical namespace (HNS) enabled accounts (over DFS endpoint). This PR adds support for User Delegation SAS for non-HNS accounts over Blob endpoint as well.
Refer: https://hadoop.apache.org/docs/current/hadoop-azure/abfs.html#Shared_Access_Signature_.28SAS.29_Token_Provider
Tests added in comments below.