HDDS-1660 Use Picocli for Ozone Manager #925

sodonnel · 2019-06-07T15:35:58Z

Replicate the changes made in HDDS-1622 for the StorageContainerManager to the Ozone Manager, so it also uses Picocli for the command line interface.

sodonnel · 2019-06-07T15:36:22Z

/label ozone

hadoop-yetus · 2019-06-07T17:51:22Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	37	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
0	shelldocs	1	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	47	Maven dependency ordering for branch
+1	mvninstall	545	trunk passed
+1	compile	300	trunk passed
+1	checkstyle	93	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	812	branch has no errors when building and testing our client artifacts.
+1	javadoc	184	trunk passed
0	spotbugs	386	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	592	trunk passed
		_ Patch Compile Tests _
0	mvndep	28	Maven dependency ordering for patch
+1	mvninstall	493	the patch passed
+1	compile	312	the patch passed
+1	javac	312	the patch passed
+1	checkstyle	96	the patch passed
+1	mvnsite	0	the patch passed
+1	shellcheck	27	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	3	The patch has no ill-formed XML file.
+1	shadedclient	678	patch has no errors when building and testing our client artifacts.
+1	javadoc	192	the patch passed
+1	findbugs	594	the patch passed
		_ Other Tests _
-1	unit	184	hadoop-hdds in the patch failed.
-1	unit	2465	hadoop-ozone in the patch failed.
+1	asflicense	72	The patch does not generate ASF License warnings.
		8059

Reason	Tests
Failed junit tests	hadoop.ozone.container.common.impl.TestHddsDispatcher
	hadoop.ozone.client.rpc.TestReadRetries
	hadoop.ozone.client.rpc.TestContainerStateMachine
	hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
	hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.hdds.scm.pipeline.TestSCMPipelineManager
	hadoop.ozone.om.TestOzoneManagerHA
	hadoop.ozone.TestSecureOzoneCluster
	hadoop.ozone.web.client.TestBuckets

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-925/1/artifact/out/Dockerfile
GITHUB PR	#925
Optional Tests	dupname asflicense mvnsite unit shellcheck shelldocs xml compile javac javadoc mvninstall shadedclient findbugs checkstyle
uname	Linux 22430c5e1af7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `14552d1`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/1/testReport/
Max. process+thread count	4204 (vs. ulimit of 5500)
modules	C: hadoop-ozone hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test hadoop-ozone/tools U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/1/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-06-07T21:34:01Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	44	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	17	Maven dependency ordering for branch
+1	mvninstall	526	trunk passed
+1	compile	316	trunk passed
+1	checkstyle	94	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	885	branch has no errors when building and testing our client artifacts.
+1	javadoc	187	trunk passed
0	spotbugs	349	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	547	trunk passed
		_ Patch Compile Tests _
0	mvndep	27	Maven dependency ordering for patch
+1	mvninstall	492	the patch passed
+1	compile	356	the patch passed
+1	javac	356	the patch passed
+1	checkstyle	125	the patch passed
+1	mvnsite	0	the patch passed
+1	shellcheck	26	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	3	The patch has no ill-formed XML file.
+1	shadedclient	771	patch has no errors when building and testing our client artifacts.
+1	javadoc	185	the patch passed
+1	findbugs	564	the patch passed
		_ Other Tests _
-1	unit	201	hadoop-hdds in the patch failed.
-1	unit	1546	hadoop-ozone in the patch failed.
+1	asflicense	62	The patch does not generate ASF License warnings.
		7279

Reason	Tests
Failed junit tests	hadoop.ozone.container.common.impl.TestHddsDispatcher
	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
	hadoop.ozone.client.rpc.TestWatchForCommit
	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
	hadoop.hdds.scm.pipeline.TestRatisPipelineProvider

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-925/2/artifact/out/Dockerfile
GITHUB PR	#925
Optional Tests	dupname asflicense mvnsite unit shellcheck shelldocs xml compile javac javadoc mvninstall shadedclient findbugs checkstyle
uname	Linux e74689f0819e 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 4e38daf
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/2/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/2/testReport/
Max. process+thread count	4285 (vs. ulimit of 5500)
modules	C: hadoop-ozone hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test hadoop-ozone/tools U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/2/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-06-11T21:06:22Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	508	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	50	Maven dependency ordering for branch
+1	mvninstall	524	trunk passed
+1	compile	284	trunk passed
+1	checkstyle	84	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	752	branch has no errors when building and testing our client artifacts.
+1	javadoc	184	trunk passed
0	spotbugs	338	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	531	trunk passed
		_ Patch Compile Tests _
0	mvndep	30	Maven dependency ordering for patch
+1	mvninstall	486	the patch passed
+1	compile	285	the patch passed
+1	javac	285	the patch passed
+1	checkstyle	79	the patch passed
+1	mvnsite	0	the patch passed
+1	shellcheck	24	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	626	patch has no errors when building and testing our client artifacts.
+1	javadoc	170	the patch passed
+1	findbugs	538	the patch passed
		_ Other Tests _
-1	unit	163	hadoop-hdds in the patch failed.
-1	unit	964	hadoop-ozone in the patch failed.
+1	asflicense	68	The patch does not generate ASF License warnings.
		6661

Reason	Tests
Failed junit tests	hadoop.ozone.container.common.impl.TestHddsDispatcher
	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.ozone.client.rpc.TestOzoneAtRestEncryption

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-925/3/artifact/out/Dockerfile
GITHUB PR	#925
Optional Tests	dupname asflicense mvnsite unit shellcheck shelldocs xml compile javac javadoc mvninstall shadedclient findbugs checkstyle
uname	Linux f69c6547787b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `5740eea`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/3/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/3/testReport/
Max. process+thread count	5323 (vs. ulimit of 5500)
modules	C: hadoop-ozone hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test hadoop-ozone/tools U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/3/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

… class when running OM commands

…anagerStarter script

…use the new method spec

…ng secure test failures.

…as causing secure test failures." This is because it appears to be the test which is broken rather than the refactored code. This reverts commit 021796b964ba05edd3c7458dad96b4f2ed2c919f.

hadoop-yetus · 2019-06-13T13:19:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	863	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 6 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	47	Maven dependency ordering for branch
+1	mvninstall	602	trunk passed
+1	compile	303	trunk passed
+1	checkstyle	97	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	852	branch has no errors when building and testing our client artifacts.
+1	javadoc	192	trunk passed
0	spotbugs	351	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	558	trunk passed
		_ Patch Compile Tests _
0	mvndep	26	Maven dependency ordering for patch
+1	mvninstall	492	the patch passed
+1	compile	312	the patch passed
+1	javac	312	the patch passed
+1	checkstyle	106	the patch passed
+1	mvnsite	0	the patch passed
+1	shellcheck	25	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	3	The patch has no ill-formed XML file.
+1	shadedclient	693	patch has no errors when building and testing our client artifacts.
+1	javadoc	179	the patch passed
+1	findbugs	563	the patch passed
		_ Other Tests _
-1	unit	236	hadoop-hdds in the patch failed.
-1	unit	1269	hadoop-ozone in the patch failed.
+1	asflicense	54	The patch does not generate ASF License warnings.
		7785

Reason	Tests
Failed junit tests	hadoop.ozone.container.ozoneimpl.TestOzoneContainer
	hadoop.ozone.container.common.impl.TestHddsDispatcher
	hadoop.hdds.scm.pipeline.TestNodeFailure
	hadoop.hdds.scm.pipeline.TestRatisPipelineProvider
	hadoop.hdds.scm.pipeline.TestNode2PipelineMap

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-925/4/artifact/out/Dockerfile
GITHUB PR	#925
Optional Tests	dupname asflicense mvnsite unit shellcheck shelldocs xml compile javac javadoc mvninstall shadedclient findbugs checkstyle
uname	Linux 6f71d6a9aa1c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `940bcf0`
Default Java	1.8.0_212
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/4/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/4/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/4/testReport/
Max. process+thread count	1955 (vs. ulimit of 5500)
modules	C: hadoop-ozone hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test hadoop-ozone/tools U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-925/4/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

sodonnel · 2019-06-17T10:23:04Z

There are two failing tests:

org.apache.hadoop.ozone.client.rpc.TestBCSID.org.apache.hadoop.ozone.client.rpc.TestBCSID

This passes locally on trunk and on the branch with this change.

org.apache.hadoop.hdds.scm.node.TestNodeReportHandler.testNodeReport

This fails on trunk and on branch. I believe the test is broken by HDDS-1663, as it added the line:

this.clusterMap = scmManager.getClusterMap();

And the test passes a null scmManager into the constructor, leading to the null pointer exception.

elek

+1

Thanks @sodonnel to fix this.

Tested and worked well. I can confirm that the unit test failures are independent.

Will merge it soon.

Closes apache#925

Author: Jagadish <[email protected]> Reviewers: Shanthoosh <[email protected]> Closes apache#925 from vjagadish1989/samza-2109

elek added the ozone label Jun 7, 2019

sodonnel force-pushed the HDDS-1660 branch from e7495ae to dca0428 Compare June 7, 2019 19:31

sodonnel force-pushed the HDDS-1660 branch from dca0428 to 021796b Compare June 11, 2019 19:14

S O'Donnell added 7 commits June 13, 2019 12:06

Change the Ozone Manager CLI startup / init code to use Picocli

29b5a28

Change the ozone executable script to run the new OzoneManagerStarter…

daff579

… class when running OM commands

Changed the dev-support Intellij configurations to use the new OzoneM…

ca9909f

…anagerStarter script

Changed reference to the old OzoneManager.createOm in GenesisUtil to …

2c3b134

…use the new method spec

Removed the loginIfSecurityEnabled method from omInit as it was causi…

8ad4c71

…ng secure test failures.

Revert "Removed the loginIfSecurityEnabled method from omInit as it w…

538d12f

…as causing secure test failures." This is because it appears to be the test which is broken rather than the refactored code. This reverts commit 021796b964ba05edd3c7458dad96b4f2ed2c919f.

Added workaround for failing test and comment explaining why

e7e5bfa

sodonnel force-pushed the HDDS-1660 branch from 021796b to e7e5bfa Compare June 13, 2019 11:07

elek approved these changes Jun 17, 2019

View reviewed changes

elek closed this in 22b36dd Jun 17, 2019

bshashikant pushed a commit to bshashikant/hadoop that referenced this pull request Jul 10, 2019

HDDS-1660. Use Picocli for Ozone Manager

b5fa231

Closes apache#925

shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019

SAMZA-2109: Reduce per-partition default buffer sizes

c9083e8

Author: Jagadish <[email protected]> Reviewers: Shanthoosh <[email protected]> Closes apache#925 from vjagadish1989/samza-2109

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-1660 Use Picocli for Ozone Manager #925

HDDS-1660 Use Picocli for Ozone Manager #925

Uh oh!

sodonnel commented Jun 7, 2019

Uh oh!

sodonnel commented Jun 7, 2019

Uh oh!

hadoop-yetus commented Jun 7, 2019

Uh oh!

hadoop-yetus commented Jun 7, 2019

Uh oh!

hadoop-yetus commented Jun 11, 2019

Uh oh!

hadoop-yetus commented Jun 13, 2019

Uh oh!

sodonnel commented Jun 17, 2019

Uh oh!

elek left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDDS-1660 Use Picocli for Ozone Manager #925

HDDS-1660 Use Picocli for Ozone Manager #925

Uh oh!

Conversation

sodonnel commented Jun 7, 2019

Uh oh!

sodonnel commented Jun 7, 2019

Uh oh!

hadoop-yetus commented Jun 7, 2019

Uh oh!

hadoop-yetus commented Jun 7, 2019

Uh oh!

hadoop-yetus commented Jun 11, 2019

Uh oh!

hadoop-yetus commented Jun 13, 2019

Uh oh!

sodonnel commented Jun 17, 2019

Uh oh!

elek left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants