Skip to content

HADOOP-18547. Check if config value is not empty string in AbfsConfiguration.getMandatoryPasswordString() #5177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

HADOOP-18547. Check if config value is not empty string in AbfsConfiguration.getMandatoryPasswordString() #5177

wants to merge 3 commits into from

Conversation

@saxenapranav
Copy link
Contributor

@saxenapranav saxenapranav commented Dec 1, 2022 •
edited
Loading

JIRA: https://issues.apache.org/jira/browse/HADOOP-18547.
The method getMandatoryPasswordString is called in `AbfsConfiguration.getTokenProvider()' to check if following configs are non-null (diff keys applicable for different implementation of AccessTokenProvider):

  1. fs.azure.account.oauth2.client.endpoint: in ClientCredsTokenProvider
  2. fs.azure.account.oauth2.client.id: in ClientCredsTokenProvider, MsiTokenProvider, RefreshTokenBasedTokenProvider
  3. fs.azure.account.oauth2.client.secret: in ClientCredsTokenProvider
  4. fs.azure.account.oauth2.client.endpoint: in UserPasswordTokenProvider
  5. fs.azure.account.oauth2.user.name: in UserPasswordTokenProvider
  6. fs.azure.account.oauth2.user.password: in UserPasswordTokenProvider
  7. fs.azure.account.oauth2.msi.tenant: in MsiTokenProvider
  8. fs.azure.account.oauth2.refresh.token: in RefreshTokenBasedTokenProvider

Right now, this method checks if its non-null and not non-empty. This PR adds check on non-empty config values.

Also fixing below test-failure as part of pr:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"

Testing

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] TestExponentialRetryPolicy.testOperationOnAccountIdle:216 » AccessDenied Opera...
[INFO]
[ERROR] Tests run: 112, Failures: 0, Errors: 1, Skipped: 1
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataContributor:84 » AccessDenied Operat...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataReader:143 » AccessDenied Operation ...
[INFO]
[ERROR] Tests run: 568, Failures: 0, Errors: 3, Skipped: 99
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_120_terasort:262->executeStage:206 » IO The ownership o...
[INFO]
[ERROR] Tests run: 336, Failures: 0, Errors: 1, Skipped: 55

HNS-SharedKey

[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] TestExponentialRetryPolicy.testOperationOnAccountIdle:216 » AccessDenied Opera...
[INFO]
[ERROR] Tests run: 112, Failures: 0, Errors: 1, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 568, Failures: 0, Errors: 1, Skipped: 54
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpUpdateCheckFileSkip:919->AbstractContractDistCpTest.verifySkipAndCopyCounter:1000->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in COPY counter value expected:<1> but was:<0>
[INFO]
[ERROR] Tests run: 336, Failures: 1, Errors: 0, Skipped: 41

NonHNS-SharedKey

[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] TestExponentialRetryPolicy.testOperationOnAccountIdle:216 » AccessDenied Opera...
[INFO]
[ERROR] Tests run: 112, Failures: 0, Errors: 1, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:336 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 568, Failures: 0, Errors: 1, Skipped: 277
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 teragen(1000, abfs://[email protected]/ITestAbfsTerasort/sortin) failed expected:<0> but was:<1>
[ERROR] ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpUpdateCheckFileSkip:919->AbstractContractDistCpTest.verifySkipAndCopyCounter:1000->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in COPY counter value expected:<1> but was:<0>
[ERROR] Errors:
[ERROR] ITestAbfsJobThroughManifestCommitter.test_0420_validateJob » OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitLifecycle » OutputValidation abfs:/... [ERROR] ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter » OutputValidation ... [ERROR] ITestAbfsManifestCommitProtocol.testOutputFormatIntegration » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit » OutputValidation ...
[INFO]
[ERROR] Tests run: 336, Failures: 2, Errors: 8, Skipped: 46

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] TestExponentialRetryPolicy.testOperationOnAccountIdle:216 » AccessDenied Opera...
[INFO]
[ERROR] Tests run: 112, Failures: 0, Errors: 1, Skipped: 1
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataContributor:84 » AccessDenied Operat...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataReader:143 » AccessDenied Operation ...
[INFO]
[ERROR] Tests run: 568, Failures: 0, Errors: 3, Skipped: 99
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsFileSystemContractAppend>AbstractContractAppendTest.testAppendNonexistentFile:85->Assert.fail:89 expected a failure
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_120_terasort:262->executeStage:206 » IO The ownership o...
[ERROR] ITestAbfsFileSystemContractAppend>AbstractContractAppendTest.testAppendToExistingFile:96 » FileAlreadyExists
[INFO]
[ERROR] Tests run: 336, Failures: 1, Errors: 2, Skipped: 55

@hadoop-yetus
Copy link

hadoop-yetus commented Dec 1, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 43s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 40m 25s trunk passed
+1 💚 compile 0m 51s trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚 compile 0m 48s trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 checkstyle 0m 43s trunk passed
+1 💚 mvnsite 0m 52s trunk passed
-1 ❌ javadoc 0m 47s /branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt hadoop-azure in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚 javadoc 0m 37s trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 spotbugs 1m 23s trunk passed
+1 💚 shadedclient 22m 49s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 36s the patch passed
+1 💚 compile 0m 36s the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚 javac 0m 36s the patch passed
+1 💚 compile 0m 33s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 javac 0m 33s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 24s the patch passed
+1 💚 mvnsite 0m 39s the patch passed
-1 ❌ javadoc 0m 31s /patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt hadoop-azure in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚 javadoc 0m 28s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 spotbugs 1m 13s the patch passed
+1 💚 shadedclient 22m 27s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 6s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 42s The patch does not generate ASF License warnings.
101m 41s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5177/1/artifact/out/Dockerfile
GITHUB PR #5177
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 16795c941670 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / b7b2633
Default Java Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5177/1/testReport/
Max. process+thread count 728 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5177/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented Dec 2, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 37s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 38m 57s trunk passed
+1 💚 compile 0m 59s trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚 compile 0m 55s trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 checkstyle 0m 52s trunk passed
+1 💚 mvnsite 1m 1s trunk passed
-1 ❌ javadoc 0m 59s /branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt hadoop-azure in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚 javadoc 0m 50s trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 spotbugs 1m 34s trunk passed
+1 💚 shadedclient 22m 39s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 39s the patch passed
+1 💚 compile 0m 42s the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚 javac 0m 42s the patch passed
+1 💚 compile 0m 36s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 javac 0m 36s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 26s the patch passed
+1 💚 mvnsite 0m 41s the patch passed
-1 ❌ javadoc 0m 31s /patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt hadoop-azure in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚 javadoc 0m 31s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 spotbugs 1m 15s the patch passed
+1 💚 shadedclient 21m 41s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 13s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 50s The patch does not generate ASF License warnings.
100m 45s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5177/2/artifact/out/Dockerfile
GITHUB PR #5177
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 1969123ed4bd 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 2959a9c
Default Java Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5177/2/testReport/
Max. process+thread count 671 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5177/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor

steveloughran commented Feb 9, 2023

LGTM, if its ready for review

@hadoop-yetus
Copy link

hadoop-yetus commented Feb 10, 2023

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 7s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 47m 8s trunk passed
+1 💚 compile 0m 39s trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚 compile 0m 35s trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 checkstyle 0m 30s trunk passed
+1 💚 mvnsite 0m 40s trunk passed
-1 ❌ javadoc 0m 37s /branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt hadoop-azure in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚 javadoc 0m 29s trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 spotbugs 1m 14s trunk passed
+1 💚 shadedclient 26m 40s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 37s the patch passed
+1 💚 compile 0m 32s the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚 javac 0m 32s the patch passed
+1 💚 compile 0m 28s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 javac 0m 28s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 17s the patch passed
+1 💚 mvnsite 0m 31s the patch passed
-1 ❌ javadoc 0m 23s /patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt hadoop-azure in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚 javadoc 0m 22s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 spotbugs 1m 4s the patch passed
+1 💚 shadedclient 27m 9s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 4s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 33s The patch does not generate ASF License warnings.
114m 48s
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5177/3/artifact/out/Dockerfile
GITHUB PR #5177
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 12a4ab51a07b 4.15.0-197-generic #208-Ubuntu SMP Tue Nov 1 17:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1ae02b4
Default Java Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5177/3/testReport/
Max. process+thread count 613 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5177/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@saxenapranav saxenapranav marked this pull request as ready for review February 13, 2023 07:23
@saxenapranav
Copy link
Contributor Author

saxenapranav commented Feb 13, 2023

LGTM, if its ready for review

@steveloughran , requesting you for your kind review. Thanks.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 30, 2025

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.

@github-actions github-actions bot added the Stale label Oct 30, 2025
@github-actions github-actions bot closed this Oct 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@saxenapranav @hadoop-yetus @steveloughran