HADOOP-17723. [build] fix the Dockerfile for ARM #3037

jojochuang · 2021-05-21T09:18:24Z

verified manually by running
dev-support/bin/create-release --asfrelease --docker --dockercache on an ARM machine.

jojochuang · 2021-05-21T09:18:57Z

@aajisaka @brahmareddybattula @vinayakumarb could you review the PR?

hadoop-yetus · 2021-05-21T09:21:01Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3037/1/console in case of problems.

jojochuang · 2021-05-21T09:38:34Z

This is broken by #1738

jojochuang · 2021-05-21T09:45:38Z

WIP. Please wait for me to run through the whole release creation process for ARM.

hadoop-yetus · 2021-05-21T10:05:15Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 57s		Maven dependency ordering for branch
+1 💚	shadedclient	13m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s		Maven dependency ordering for patch
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	hadolint	0m 1s		No new issues.
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	12m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		45m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3037/1/artifact/out/Dockerfile
GITHUB PR	#3037
Optional Tests	dupname asflicense codespell hadolint shellcheck shelldocs
uname	Linux 42e29ca53150 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / dd01ced464bb85bfe9fef5657e2872a1e5618474
Max. process+thread count	680 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3037/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

busbey · 2021-05-21T14:18:44Z

This is broken by #1738

That was awhile ago. Gap in our CI coverage for ARM?

aajisaka · 2021-05-21T15:33:43Z

There exists a daily ARM qbt job, but it does not use the docker image: https://ci-hadoop.apache.org/job/hadoop-qbt-linux-ARM-trunk/

jojochuang · 2021-05-21T15:36:44Z

This is broken by #1738

That was awhile ago. Gap in our CI coverage for ARM?

I guess the ARM nightly build doesn't use the docker image? Or it has been failing for a while? Because apparently the python3 migration wasn't successful. Other than this, yarn-ui profile still requires python2. Additionally, part of the code requires phantomJS where the ARM binary was built for Ubuntu Xenial, not Bionic (missing dynamic libs only available in Xenial.)

busbey · 2021-05-21T20:17:29Z

Ah. Yeah looks like despite being named with the pattern we use for Yetus driven CI, the arm job is just a bash script wrapping a maven call.

Change-Id: I8a2031ee1e532f5ecbf32d679d850be4fb80c8d1

hadoop-yetus · 2021-05-22T02:44:32Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3037/2/console in case of problems.

jojochuang · 2021-05-22T02:57:17Z

Installs python2.7, and replaced the custom PhantomJS binary (which was built for Xenial and does not run on Bionic) with the official phantomjs package for Bionic.

Note: Yes I am aware we migrated to Python3. However, the installation of nodejs implicitly installs python-minimal, which is python2 without lots of packages. So python2 is installed by the ARM Docker image anyway (possibly the x86 docker image has it too).

The official Ubuntu phantomjs has a caveat: it requires X Window. (ariya/phantomjs#14376 (comment)). Add environment var QT_QPA_PLATFORM=offscreen to make it really 'headless". If it's not headless, the installer fails to execute phantomjs and abort.

In comparision, the x86 Dockerfile doesn't install the official Ubuntu phantomjs package. Instead, it installs the version managed by the npm/yarn package manager.

We need python2.7 package, because hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui installs node-gyp 3.8, which has dependency on python2 to build. Not clear to me why the x86 Dockerfile doesn't require it. My guess is that the node-gyp has official x86 binary so it doesn't need to be built from source.

That's a lot of digest in this comment since it involves Docker, Ubuntu, Maven, yarn, Python. Lots of different package managers. But at the end of it I was able to build for ARM.

jojochuang · 2021-05-22T02:59:55Z

dev-support/docker/Dockerfile_aarch64

        python3-pkg-resources \
        python3-setuptools \
        python3-wheel \
+        python2.7 \


We need python2.7 package, because hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui installs node-gyp 3.8, which has dependency on python2 to build.

jojochuang · 2021-05-22T03:00:21Z

dev-support/docker/Dockerfile_aarch64

 # hadolint ignore=DL3008
 RUN npm install -g [email protected]

-###


replaced the custom PhantomJS binary (which was built for Xenial and does not run on Bionic) with the official phantomjs package for Bionic.

jojochuang · 2021-05-22T03:00:40Z

dev-support/docker/Dockerfile_aarch64


+# Force PhantomJS to be in 'headless' mode, do not connect to Xwindow
+ENV QT_QPA_PLATFORM offscreen
+


The official Ubuntu phantomjs has a caveat: it requires X Window. (ariya/phantomjs#14376 (comment)). Add environment var QT_QPA_PLATFORM=offscreen to make it really 'headless". If it's not headless, the installer fails to execute phantomjs and abort.

hadoop-yetus · 2021-05-22T03:37:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	8m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 13s		Maven dependency ordering for branch
+1 💚	shadedclient	13m 8s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	hadolint	0m 3s		No new issues.
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	12m 48s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		53m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3037/2/artifact/out/Dockerfile
GITHUB PR	#3037
Optional Tests	dupname asflicense codespell hadolint shellcheck shelldocs
uname	Linux 6d5abe4bbfd0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `6354405`
Max. process+thread count	720 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3037/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

dev-support/docker/Dockerfile_aarch64

aajisaka

I don't have an ARM machine to try with.
However, I'm +1 if the order of the package list is addressed.

Could you try the docker image? @brahmareddybattula @ayushtkn

Change-Id: I1e38d8d4da24922e7a1a36f917203ca0d45c4a13

hadoop-yetus · 2021-05-24T08:27:34Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3037/3/console in case of problems.

hadoop-yetus · 2021-05-24T09:11:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 7s		Maven dependency ordering for branch
+1 💚	shadedclient	13m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	hadolint	0m 2s		No new issues.
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	12m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		45m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3037/3/artifact/out/Dockerfile
GITHUB PR	#3037
Optional Tests	dupname asflicense codespell hadolint shellcheck shelldocs
uname	Linux b5eb6dd19d2f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `35ea183`
Max. process+thread count	658 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3037/3/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

vinayakumarb · 2021-05-24T09:17:57Z

Ah. Yeah looks like despite being named with the pattern we use for Yetus driven CI, the arm job is just a bash script wrapping a maven call.

The reason for this was, at that time, yetus was not supporting aarch64 CIs. Not sure whether latest Yetus supports aarch64.

jojochuang · 2021-05-24T10:20:34Z

Merged. Will cherrypick to branch-3.3.1. Thanks @aajisaka, @busbey and @vinayakumarb for review and inputs.

Reviewed-by: Akira Ajisaka <[email protected]> (cherry picked from commit 1576f81)

Reviewed-by: Akira Ajisaka <[email protected]> (cherry picked from commit 1576f81) (cherry picked from commit 745a2f5)

iwasakims · 2021-05-24T11:11:31Z

Thanks, @jojochuang. I got no issue on building and testing of yarn-ui on EC2 A1 instance with the patch.

Reviewed-by: Akira Ajisaka <[email protected]>

jojochuang requested review from brahmareddybattula, aajisaka and vinayakumarb May 21, 2021 09:18

HADOOP-17723. [build] fix the Dockerfile for ARM

6354405

Change-Id: I8a2031ee1e532f5ecbf32d679d850be4fb80c8d1

jojochuang force-pushed the HADOOP-17723 branch from dd01ced to 6354405 Compare May 22, 2021 02:41

jojochuang commented May 22, 2021

View reviewed changes

jojochuang changed the title ~~HADOOP-17723. [build] python3 migration broke Docker container image for ARM.~~ HADOOP-17723. [build] fix the Dockerfile for ARM May 22, 2021

jojochuang mentioned this pull request May 22, 2021

BIGTOP-3526: Remove phantomjs dependency from yarn-ui apache/bigtop#759

Merged

aajisaka reviewed May 24, 2021

View reviewed changes

dev-support/docker/Dockerfile_aarch64 Outdated Show resolved Hide resolved

aajisaka reviewed May 24, 2021

View reviewed changes

Sort packges by alphabetical order.

35ea183

Change-Id: I1e38d8d4da24922e7a1a36f917203ca0d45c4a13

aajisaka approved these changes May 24, 2021

View reviewed changes

jojochuang merged commit 1576f81 into apache:trunk May 24, 2021

asfgit pushed a commit that referenced this pull request May 24, 2021

HADOOP-17723. [build] fix the Dockerfile for ARM (#3037)

745a2f5

Reviewed-by: Akira Ajisaka <[email protected]> (cherry picked from commit 1576f81)

asfgit pushed a commit that referenced this pull request May 24, 2021

HADOOP-17723. [build] fix the Dockerfile for ARM (#3037)

1aaec69

Reviewed-by: Akira Ajisaka <[email protected]> (cherry picked from commit 1576f81) (cherry picked from commit 745a2f5)

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HADOOP-17723. [build] fix the Dockerfile for ARM (apache#3037)

a534fad

Reviewed-by: Akira Ajisaka <[email protected]>


		# Force PhantomJS to be in 'headless' mode, do not connect to Xwindow
		ENV QT_QPA_PLATFORM offscreen

HADOOP-17723. [build] fix the Dockerfile for ARM #3037

HADOOP-17723. [build] fix the Dockerfile for ARM #3037

Uh oh!

Conversation

jojochuang commented May 21, 2021

Uh oh!

jojochuang commented May 21, 2021

Uh oh!

hadoop-yetus commented May 21, 2021

Uh oh!

jojochuang commented May 21, 2021

Uh oh!

jojochuang commented May 21, 2021

Uh oh!

hadoop-yetus commented May 21, 2021

Uh oh!

busbey commented May 21, 2021

Uh oh!

aajisaka commented May 21, 2021

Uh oh!

jojochuang commented May 21, 2021

Uh oh!

busbey commented May 21, 2021

Uh oh!

hadoop-yetus commented May 22, 2021

Uh oh!

jojochuang commented May 22, 2021

Uh oh!

jojochuang May 22, 2021

Choose a reason for hiding this comment

Uh oh!

jojochuang May 22, 2021

Choose a reason for hiding this comment

Uh oh!

jojochuang May 22, 2021

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented May 22, 2021

Uh oh!

Uh oh!

aajisaka left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented May 24, 2021

Uh oh!

hadoop-yetus commented May 24, 2021

Uh oh!

vinayakumarb commented May 24, 2021

Uh oh!

jojochuang commented May 24, 2021

Uh oh!

iwasakims commented May 24, 2021

Uh oh!

Uh oh!