Skip to content

HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. #2138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Nov 11, 2020
Merged

HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. #2138

merged 1 commit into from
Nov 11, 2020

Conversation

@jianghuazhu
Copy link
Contributor

@jianghuazhu jianghuazhu commented Jul 14, 2020

…ET_SIZE.

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

@hadoop-yetus
Copy link

hadoop-yetus commented Jul 14, 2020

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 21m 45s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 1m 6s Maven dependency ordering for branch
+1 💚 mvninstall 19m 14s trunk passed
+1 💚 compile 3m 54s trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚 compile 3m 31s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 checkstyle 0m 59s trunk passed
+1 💚 mvnsite 2m 12s trunk passed
+1 💚 shadedclient 17m 7s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 40s hadoop-hdfs-client in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
-1 ❌ javadoc 0m 37s hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚 javadoc 1m 23s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗 spotbugs 2m 57s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 5m 8s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 26s Maven dependency ordering for patch
+1 💚 mvninstall 1m 53s the patch passed
+1 💚 compile 3m 45s the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚 javac 3m 45s the patch passed
+1 💚 compile 3m 27s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 javac 3m 27s the patch passed
-0 ⚠️ checkstyle 0m 51s hadoop-hdfs-project: The patch generated 2 new + 35 unchanged - 0 fixed = 37 total (was 35)
+1 💚 mvnsite 1m 57s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 shadedclient 13m 43s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 34s hadoop-hdfs-client in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
-1 ❌ javadoc 0m 32s hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚 javadoc 1m 15s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 findbugs 5m 16s the patch passed
_ Other Tests _
+1 💚 unit 2m 1s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 94m 41s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 41s The patch does not generate ASF License warnings.
209m 46s
Reason Tests
Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAAppend
hadoop.hdfs.TestGetFileChecksum
hadoop.hdfs.TestStripedFileAppend
hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2138/1/artifact/out/Dockerfile
GITHUB PR #2138
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname Linux ee8c074113cd 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / bdce75d
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2138/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2138/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2138/1/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2138/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-2138/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2138/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2138/1/testReport/
Max. process+thread count 4698 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2138/1/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

sunchao
sunchao approved these changes Jul 24, 2020
View reviewed changes
Copy link
Member

@sunchao sunchao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

...ect/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
"dfs.encrypt.data.transfer.cipher.key.bitlength";
int DFS_ENCRYPT_DATA_TRANSFER_CIPHER_KEY_BITLENGTH_DEFAULT = 128;

public static final String
Copy link
Member

@sunchao sunchao Jul 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: fix the checkstyle complain

@hadoop-yetus
Copy link

hadoop-yetus commented Jul 31, 2020

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 17s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 0m 29s Maven dependency ordering for branch
+1 💚 mvninstall 23m 12s trunk passed
+1 💚 compile 4m 25s trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 compile 4m 4s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 checkstyle 1m 0s trunk passed
+1 💚 mvnsite 2m 19s trunk passed
+1 💚 shadedclient 19m 49s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 38s hadoop-hdfs-client in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
-1 ❌ javadoc 0m 37s hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚 javadoc 1m 18s trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗 spotbugs 3m 22s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 5m 53s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 27s Maven dependency ordering for patch
+1 💚 mvninstall 2m 3s the patch passed
+1 💚 compile 4m 22s the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚 javac 4m 22s the patch passed
+1 💚 compile 3m 59s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 javac 3m 59s the patch passed
-0 ⚠️ checkstyle 0m 54s hadoop-hdfs-project: The patch generated 2 new + 35 unchanged - 0 fixed = 37 total (was 35)
+1 💚 mvnsite 1m 58s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 shadedclient 16m 32s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 33s hadoop-hdfs-client in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
-1 ❌ javadoc 0m 29s hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚 javadoc 1m 4s the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚 findbugs 5m 57s the patch passed
_ Other Tests _
+1 💚 unit 2m 9s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 1m 16s hadoop-hdfs in the patch failed.
+1 💚 asflicense 0m 31s The patch does not generate ASF License warnings.
107m 58s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/3/artifact/out/Dockerfile
GITHUB PR #2138
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname Linux cb14534b125b 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 05b3337
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/3/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/3/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
checkstyle https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/3/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/3/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/3/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
unit https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/3/testReport/
Max. process+thread count 307 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/3/console
versions git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@jojochuang
Copy link
Contributor

jojochuang commented Nov 10, 2020

looks almost good barring some nitty checkstyle warnings. Don't think the javadoc errors are related but i'll kick off a rebuild to verify.

@sunchao
Copy link
Member

sunchao commented Nov 10, 2020

Oops sorry I forgot to commit this. Thanks @jojochuang for picking it up!

@hadoop-yetus
Copy link

hadoop-yetus commented Nov 11, 2020

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 29s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 0m 0s test4tests The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 59s Maven dependency ordering for branch
+1 💚 mvninstall 20m 33s trunk passed
+1 💚 compile 4m 5s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 compile 3m 48s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 checkstyle 1m 0s trunk passed
+1 💚 mvnsite 2m 16s trunk passed
+1 💚 shadedclient 18m 26s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 34s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 2m 3s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+0 🆗 spotbugs 3m 3s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 5m 28s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 29s Maven dependency ordering for patch
+1 💚 mvninstall 2m 1s the patch passed
+1 💚 compile 3m 59s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javac 3m 59s the patch passed
+1 💚 compile 3m 36s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 javac 3m 36s the patch passed
-0 ⚠️ checkstyle 0m 50s /diff-checkstyle-hadoop-hdfs-project.txt hadoop-hdfs-project: The patch generated 2 new + 35 unchanged - 0 fixed = 37 total (was 35)
+1 💚 mvnsite 1m 59s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 shadedclient 14m 54s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 24s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 1m 52s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 findbugs 5m 34s the patch passed
_ Other Tests _
+1 💚 unit 2m 20s hadoop-hdfs-client in the patch passed.
+1 💚 unit 96m 33s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 40s The patch does not generate ASF License warnings.
211m 53s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/6/artifact/out/Dockerfile
GITHUB PR #2138
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname Linux e79decd213af 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / fbd2220
Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/6/testReport/
Max. process+thread count 4522 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2138/6/console
versions git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@sunchao
Copy link
Member

sunchao commented Nov 11, 2020

Test results looks good. Merging to master.

@sunchao sunchao merged commit e6d2dcc into apache:trunk Nov 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sunchao sunchao sunchao approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jianghuazhu @hadoop-yetus @jojochuang @sunchao