HDDS-1393. Convert all OM Bucket related operations to HA model. #704

bharatviswa504 · 2019-04-05T22:27:55Z

No description provided.

hadoop-yetus · 2019-04-06T03:47:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	25	Docker mode activated.
		_ Prechecks _
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	49	Maven dependency ordering for branch
+1	mvninstall	1023	trunk passed
+1	compile	117	trunk passed
+1	checkstyle	48	trunk passed
+1	mvnsite	114	trunk passed
+1	shadedclient	810	branch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1	findbugs	100	trunk passed
+1	javadoc	85	trunk passed
		_ Patch Compile Tests _
0	mvndep	13	Maven dependency ordering for patch
+1	mvninstall	103	the patch passed
+1	compile	104	the patch passed
+1	cc	103	the patch passed
+1	javac	103	the patch passed
+1	checkstyle	24	the patch passed
+1	mvnsite	89	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	764	patch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1	findbugs	113	the patch passed
+1	javadoc	73	the patch passed
		_ Other Tests _
+1	unit	36	common in the patch passed.
+1	unit	42	ozone-manager in the patch passed.
-1	unit	925	integration-test in the patch failed.
+1	asflicense	38	The patch does not generate ASF License warnings.
		4685

Reason	Tests
Failed junit tests	hadoop.ozone.TestMiniChaosOzoneCluster

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-704/1/artifact/out/Dockerfile
GITHUB PR	#704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname	Linux 7c3aa0c2e649 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `1b9ba0e`
maven	version: Apache Maven 3.3.9
Default Java	1.8.0_191
findbugs	v3.1.0-RC1
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/1/testReport/
Max. process+thread count	4519 (vs. ulimit of 5500)
modules	C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/1/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

hanishakoneru · 2019-04-08T22:56:53Z

hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/S3BucketManagerImpl.java

Here it should be !isRatisEnabled

Here it should be if Ratis is Enabled, we should call applyTransaction to persist this info to DB.
As we have not separated this into 2 phase we should do this.

hanishakoneru · 2019-04-08T23:06:14Z

hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto

BucketArgs and BucketInfo have a lot of common fields. Can we combine them into one proto?

Addressed for now as making bucketArgs as optional, so that we will not set this during creating OMRequest in startTransaction, and we don't write duplicate data into raft log.

Added a TODO for the suggested, as discussed offline.

bharatviswa504 · 2019-04-09T00:50:48Z

Thank You @hanishakoneru for the review.
I have addressed review comments.

hadoop-yetus · 2019-04-09T05:49:49Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	28	Docker mode activated.
		_ Prechecks _
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	42	Maven dependency ordering for branch
+1	mvninstall	1047	trunk passed
+1	compile	108	trunk passed
+1	checkstyle	47	trunk passed
+1	mvnsite	100	trunk passed
+1	shadedclient	774	branch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1	findbugs	97	trunk passed
+1	javadoc	70	trunk passed
		_ Patch Compile Tests _
0	mvndep	11	Maven dependency ordering for patch
+1	mvninstall	97	the patch passed
+1	compile	100	the patch passed
+1	cc	100	the patch passed
+1	javac	100	the patch passed
+1	checkstyle	22	the patch passed
+1	mvnsite	80	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	751	patch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1	findbugs	112	the patch passed
+1	javadoc	64	the patch passed
		_ Other Tests _
+1	unit	35	common in the patch passed.
+1	unit	37	ozone-manager in the patch passed.
-1	unit	1486	integration-test in the patch failed.
+1	asflicense	28	The patch does not generate ASF License warnings.
		5122

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
	hadoop.ozone.om.TestOzoneManagerHA

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-704/2/artifact/out/Dockerfile
GITHUB PR	#704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname	Linux 653162015ddc 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `69e3745`
maven	version: Apache Maven 3.3.9
Default Java	1.8.0_191
findbugs	v3.1.0-RC1
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/2/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/2/testReport/
Max. process+thread count	4159 (vs. ulimit of 5500)
modules	C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/2/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-04-09T05:50:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	28	Docker mode activated.
		_ Prechecks _
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	25	Maven dependency ordering for branch
+1	mvninstall	1116	trunk passed
+1	compile	112	trunk passed
+1	checkstyle	45	trunk passed
+1	mvnsite	93	trunk passed
+1	shadedclient	798	branch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1	findbugs	105	trunk passed
+1	javadoc	71	trunk passed
		_ Patch Compile Tests _
0	mvndep	10	Maven dependency ordering for patch
+1	mvninstall	102	the patch passed
+1	compile	116	the patch passed
+1	cc	116	the patch passed
+1	javac	116	the patch passed
+1	checkstyle	24	the patch passed
+1	mvnsite	88	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	717	patch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1	findbugs	111	the patch passed
+1	javadoc	72	the patch passed
		_ Other Tests _
+1	unit	34	common in the patch passed.
+1	unit	41	ozone-manager in the patch passed.
-1	unit	1387	integration-test in the patch failed.
+1	asflicense	28	The patch does not generate ASF License warnings.
		5125

Reason	Tests
Failed junit tests	hadoop.ozone.TestMiniChaosOzoneCluster
	hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
	hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
	hadoop.ozone.TestMiniOzoneCluster
	hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.ozone.om.TestOzoneManagerHA
	hadoop.ozone.om.TestScmChillMode

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-704/3/artifact/out/Dockerfile
GITHUB PR	#704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname	Linux 0152e5d203ff 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `69e3745`
maven	version: Apache Maven 3.3.9
Default Java	1.8.0_191
findbugs	v3.1.0-RC1
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/3/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/3/testReport/
Max. process+thread count	5329 (vs. ulimit of 5500)
modules	C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/3/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

bharatviswa504 · 2019-04-09T17:08:25Z

Thank You @hanishakoneru for the review.
I have addressed review comments.

hadoop-yetus · 2019-04-09T20:20:53Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	42	Docker mode activated.
		_ Prechecks _
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	13	Maven dependency ordering for branch
-1	mvninstall	1061	root in trunk failed.
-1	compile	39	hadoop-ozone in trunk failed.
+1	checkstyle	44	trunk passed
+1	mvnsite	105	trunk passed
+1	shadedclient	778	branch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1	findbugs	94	trunk passed
+1	javadoc	71	trunk passed
		_ Patch Compile Tests _
0	mvndep	9	Maven dependency ordering for patch
+1	mvninstall	90	the patch passed
-1	compile	30	hadoop-ozone in the patch failed.
-1	cc	30	hadoop-ozone in the patch failed.
-1	javac	30	hadoop-ozone in the patch failed.
+1	checkstyle	19	the patch passed
+1	mvnsite	76	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	682	patch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1	findbugs	108	the patch passed
+1	javadoc	63	the patch passed
		_ Other Tests _
+1	unit	30	common in the patch passed.
+1	unit	38	ozone-manager in the patch passed.
-1	unit	923	integration-test in the patch failed.
+1	asflicense	27	The patch does not generate ASF License warnings.
		4344

Reason	Tests
Failed junit tests	hadoop.ozone.om.TestOzoneManagerHA

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-704/4/artifact/out/Dockerfile
GITHUB PR	#704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname	Linux 8759bc92255b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `cfec455`
maven	version: Apache Maven 3.3.9
Default Java	1.8.0_191
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/4/artifact/out/branch-mvninstall-root.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/4/artifact/out/branch-compile-hadoop-ozone.txt
findbugs	v3.1.0-RC1
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/4/artifact/out/patch-compile-hadoop-ozone.txt
cc	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/4/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/4/artifact/out/patch-compile-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/4/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/4/testReport/
Max. process+thread count	4234 (vs. ulimit of 5500)
modules	C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-704/4/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

hanishakoneru · 2019-04-09T21:30:13Z

LGTM. +1 pending CI.

bharatviswa504 · 2019-04-09T21:54:48Z

Acceptance tests of S3 tests suite are passing locally, I think cluster start might not be done so all tests failed during suite test setup stage.

And test failure is not related to this patch.

I will commit this shortly.

md file for quick start. Author: xinyuiscool <[email protected]> Reviewers: Jagadish<[email protected]> Closes apache#704 from xinyuiscool/SAMZA-1911

bharatviswa504 requested review from arp7 and hanishakoneru April 5, 2019 22:27

bharatviswa504 added the ozone label Apr 5, 2019

bharatviswa504 self-assigned this Apr 5, 2019

hanishakoneru reviewed Apr 8, 2019

View reviewed changes

bharatviswa504 force-pushed the HDDS-1393 branch from 6427c22 to f5aa8ea Compare April 9, 2019 00:50

bharatviswa504 added 2 commits April 9, 2019 09:39

HDDS-1393. Convert all OM Bucket related operations to HA model.

b4205b1

fix review comments

b00e8a9

bharatviswa504 force-pushed the HDDS-1393 branch from 1708528 to b00e8a9 Compare April 9, 2019 16:58

bharatviswa504 merged commit 25c421b into apache:trunk Apr 9, 2019

HDDS-1393. Convert all OM Bucket related operations to HA model. #704

HDDS-1393. Convert all OM Bucket related operations to HA model. #704

Uh oh!

Conversation

bharatviswa504 commented Apr 5, 2019

Uh oh!

hadoop-yetus commented Apr 6, 2019

Uh oh!

hanishakoneru Apr 8, 2019

Choose a reason for hiding this comment

Uh oh!

bharatviswa504 Apr 8, 2019

Choose a reason for hiding this comment

Uh oh!

hanishakoneru Apr 8, 2019

Choose a reason for hiding this comment

Uh oh!

bharatviswa504 Apr 9, 2019

Choose a reason for hiding this comment

Uh oh!

bharatviswa504 commented Apr 9, 2019

Uh oh!

hadoop-yetus commented Apr 9, 2019

Uh oh!

hadoop-yetus commented Apr 9, 2019

Uh oh!

bharatviswa504 commented Apr 9, 2019

Uh oh!

hadoop-yetus commented Apr 9, 2019

Uh oh!

hanishakoneru commented Apr 9, 2019

Uh oh!

bharatviswa504 commented Apr 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bharatviswa504 commented Apr 9, 2019 •

edited

Loading