Merge #5
Merged
Merge #5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ls (#2431) Co-authored-by: ahussein <[email protected]>
#2255) Co-authored-by: Issac Buenrostro <[email protected]>
…reams in hdfs-default.xml. Contributed by Xieming Li.
…ted by Daryn Sharp and Ahmed Hussein
…2462) Contributed by Attila Doroszlai.
…Contributed by Ahmed Hussein
…omainNameUsingDNS fails. (#2465)
…2466) Unless you explicitly set it, the issue date of a delegation token identifier is 0, which confuses spark renewal (SPARK-33440). This patch makes sure that all S3A DT identifiers have the current time as issue date, fixing the problem as far as S3A tokens are concerned. Contributed by Jungtaek Lim.
#2310) This fixes the S3Guard/Directory Marker Retention integration so that when fs.s3a.directory.marker.retention=keep, failures during multipart delete are handled correctly, as are incremental deletes during directory tree operations. In both cases, when a directory marker with children is deleted from S3, the directory entry in S3Guard is not deleted, because it is still critical to representing the structure of the store. Contributed by Steve Loughran. Change-Id: I4ca133a23ea582cd42ec35dbf2dc85b286297d2f
…t ID. (#2399) See also [SPARK-33402]: Jobs launched in same second have duplicate MapReduce JobIDs Contributed by Steve Loughran. Change-Id: Iae65333cddc84692997aae5d902ad8765b45772a
… lenient capacity checks in Capacity Scheduler. Contributed by Peter Bacsko
…SON placement rule format. Contributed by Gergely Pollak
Contributed by Liang-Chi Hsieh.
…ntributed by Daryn Sharp and Ahmed Hussein
…le loader imp through conf (#2389). Contributed by Junfan Zhang.
Followup to HADOOP-17379. Contributed by Jungtaek Lim.
… notify the given URL via a custom class. Contributed by Zoltan Erdmann
…lt to false (#2443). Contributed by Ahmed Hussein.
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license Contributed by Zhihua Deng.
…eption. Contributed by Hongbing Wang. Reviewed-by: He Xiaoqiao <[email protected]> Reviewed-by: Hui Fei <[email protected]>
Contributed by Steve Loughran.
…2469) Contributed by Zhihua Deng.
Contributed by Ayush Saxena and Steve Loughran Change-Id: I4bb76815bc1d11d1804dc67bafde68b6a995b974
This patch * fixes the inversion * adds a precondition check * if the commands are supplied inverted, swaps them with a warning. This is to stop breaking any tests written to cope with the existing behavior. Contributed by Steve Loughran
…l should support it. Contributed by zhuqi.
Contributed by Mukund Thakur.
…apper/reducers from config. Contributed by Peter Bacsko & Wangda Tan
… fails on trunk. (#2503)
…tor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein
…mptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth)
…ontributed by Daryn Sharp and Ahmed Hussein
…2508). Contributed by Daryn Sharp and Ahmed Hussein
…n each iteration (#2483). Contributed by Leon Gao.
DETAILS:
The previous commit for HADOOP-17397 was not the correct fix. DelegationSASGenerator.getDelegationSAS
should return sp=p for the set-permission and set-acl operations. The tests have also been updated as
follows:
1. When saoid and suoid are not specified, skoid must have an RBAC role assignment which grants
Microsoft.Storage/storageAccounts/blobServices/containers/blobs/modifyPermissions/action and sp=p
to set permissions or set ACL.
2. When saoid or suiod is specified, same as 1) but furthermore the saoid or suoid must be an owner of
the file or directory in order for the operation to succeed.
3. When saoid or suiod is specified, the ownership check is bypassed by also including 'o' (ownership)
in the SAS permission (for example, sp=op). Note that 'o' grants the saoid or suoid the ability to
change the file or directory owner to themself, and they can also change the owning group. Generally
speaking, if a trusted authorizer would like to give a user the ability to change the permissions or
ACL, then that user should be the file or directory owner.
TEST RESULTS:
namespace.enabled=true
auth.type=SharedKey
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 24
Tests run: 208, Failures: 0, Errors: 0, Skipped: 24
namespace.enabled=true
auth.type=OAuth
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 70
Tests run: 208, Failures: 0, Errors: 0, Skipped: 141
Reviewed-by: Adam Antal <[email protected]> Reviewed-by: Akira Ajisaka <[email protected]>
…ion class (#2486). Contributed by Daryn Sharp and Ahmed Hussein
…ction leading to CLOSE_WAIT cnxns. Contributed by Attila Magyar. Signed-off-by: Wei-Chiu Chuang <[email protected]>
Signed-off-by: Akira Ajisaka <[email protected]> Reviewed-by: Jim Brennan <[email protected]>
…error. Contributed by HuangTao.
…c}. Contributed by Fengnan Li and CR Hota.
…storage locations. Contributed by Yang Yun.
…Contributed by Ryan Wu.
…buted by angerszhu.
Signed-off-by: Akira Ajisaka <[email protected]>
Signed-off-by: Akira Ajisaka <[email protected]>
#2494) Contributed by Qi Zhu.
…andler (#2527). Contributed by Wanqiang Ji
…d Hussein and Kihwal Lee
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
33 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.