YARN-11753. Ensure NM is marked unhealthy if the ProcessBuilder reports an issue with the container-executor #7290

brumi1024 · 2025-01-15T16:45:12Z

Description of PR

The ProcessBuilder.start method has a specific error message if the launched executable has some issues. The code introduced in YARN-11709 should match for that message instead of a generic file is missing message:

    public Process start() throws IOException {
        // Must convert to array first -- a malicious user-supplied
        // list might try to circumvent the security check.
        String[] cmdarray = command.toArray(new String[command.size()]);
        cmdarray = cmdarray.clone();

        for (String arg : cmdarray)
            if (arg == null)
                throw new NullPointerException();
        // Throws IndexOutOfBoundsException if command is empty
        String prog = cmdarray[0];

        SecurityManager security = System.getSecurityManager();
        if (security != null)
            security.checkExec(prog);

        String dir = directory == null ? null : directory.toString();

        for (String s : cmdarray) {
            if (s.indexOf('\u0000') >= 0) {
                throw new IOException("invalid null character in command");
            }
        }

        try {
            return ProcessImpl.start(cmdarray,
                                     environment,
                                     dir,
                                     redirects,
                                     redirectErrorStream);
        } catch (IOException | IllegalArgumentException e) {
            String exceptionInfo = ": " + e.getMessage();
            Throwable cause = e;
            if ((e instanceof IOException) && security != null) {
                // Can not disclose the fail reason for read-protected files.
                try {
                    security.checkRead(prog);
                } catch (SecurityException se) {
                    exceptionInfo = "";
                    cause = se;
                }
            }
            // It's much easier for us to create a high-quality error
            // message than the low-level C code which found the problem.
            throw new IOException(
                "Cannot run program \"" + prog + "\""
                + (dir == null ? "" : " (in directory \"" + dir + "\")")
                + exceptionInfo,
                cause);
        }
    }

How was this patch tested?

UTs, deployed on a cluster.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…ts an issue with the container-executor.

hadoop-yetus · 2025-01-15T18:19:52Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	22m 13s		trunk passed
+1 💚	compile	0m 47s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 50s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 24s		trunk passed
+1 💚	mvnsite	0m 26s		trunk passed
+1 💚	javadoc	0m 31s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 27s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 56s		trunk passed
+1 💚	shadedclient	19m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	0m 44s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 44s		the patch passed
+1 💚	compile	0m 42s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 16s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5)
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	javadoc	0m 21s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 20s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 53s		the patch passed
+1 💚	shadedclient	19m 40s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	22m 42s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 22s		The patch does not generate ASF License warnings.
		93m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7290/1/artifact/out/Dockerfile
GITHUB PR	#7290
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux a3e3fae0a8d3 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e8c3e84`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7290/1/testReport/
Max. process+thread count	730 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7290/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-01-16T13:45:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	26m 5s		trunk passed
+1 💚	compile	0m 53s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 23s		trunk passed
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	javadoc	0m 26s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 22s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 52s		trunk passed
+1 💚	shadedclient	22m 37s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 17s		the patch passed
+1 💚	compile	0m 51s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 51s		the patch passed
+1 💚	compile	0m 45s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 18s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5)
+1 💚	mvnsite	0m 23s		the patch passed
+1 💚	javadoc	0m 23s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 22s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 57s		the patch passed
+1 💚	shadedclient	23m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	23m 53s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 20s		The patch does not generate ASF License warnings.
		105m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7290/2/artifact/out/Dockerfile
GITHUB PR	#7290
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 5f20fe7bea32 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `716725e`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7290/2/testReport/
Max. process+thread count	559 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7290/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

p-szucs · 2025-01-16T16:53:14Z

Thanks @brumi1024, LGTM

slfan1989 · 2025-01-17T00:32:51Z

@brumi1024 Thanks for the contribution! LGTM +1. We need to fix the Checkstyle issues.

brumi1024 · 2025-01-20T13:28:10Z

Thanks @p-szucs and @slfan1989 for the review. Fixed the checkstyle issues, will be merging this to trunk once the build is successful.

hadoop-yetus · 2025-01-20T14:41:14Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	22m 38s		trunk passed
+1 💚	compile	0m 51s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 50s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 23s		trunk passed
+1 💚	mvnsite	0m 24s		trunk passed
+1 💚	javadoc	0m 27s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 24s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 51s		trunk passed
+1 💚	shadedclient	21m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 21s		the patch passed
+1 💚	compile	0m 47s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 47s		the patch passed
+1 💚	compile	0m 46s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 46s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 15s		the patch passed
+1 💚	mvnsite	0m 21s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 20s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	0m 52s		the patch passed
+1 💚	shadedclient	20m 50s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 53s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		74m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7290/3/artifact/out/Dockerfile
GITHUB PR	#7290
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 6447577c3593 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e3bccc6`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7290/3/testReport/
Max. process+thread count	556 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7290/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

…Builder reports an issue with the container-executor (apache#7290) (apache#232) Change-Id: Iaaa94c8f46faa4feaede27de36e0d94483ae0229

YARN-11753. Ensure NM is marked unhealthy if the ProcessBuilder repor…

e8c3e84

…ts an issue with the container-executor.

github-actions bot added YARN trunk labels Jan 15, 2025

Add null check.

716725e

p-szucs approved these changes Jan 16, 2025

View reviewed changes

slfan1989 approved these changes Jan 17, 2025

View reviewed changes

Fix checkstyle.

e3bccc6

brumi1024 merged commit 0d72896 into apache:trunk Jan 20, 2025
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

YARN-11753. Ensure NM is marked unhealthy if the ProcessBuilder reports an issue with the container-executor #7290

YARN-11753. Ensure NM is marked unhealthy if the ProcessBuilder reports an issue with the container-executor #7290

brumi1024 commented Jan 15, 2025 •

edited

Loading

Uh oh!

hadoop-yetus commented Jan 15, 2025

Uh oh!

hadoop-yetus commented Jan 16, 2025

Uh oh!

p-szucs commented Jan 16, 2025

Uh oh!

slfan1989 commented Jan 17, 2025

Uh oh!

brumi1024 commented Jan 20, 2025

Uh oh!

hadoop-yetus commented Jan 20, 2025

Uh oh!

Uh oh!

Uh oh!

YARN-11753. Ensure NM is marked unhealthy if the ProcessBuilder reports an issue with the container-executor #7290

YARN-11753. Ensure NM is marked unhealthy if the ProcessBuilder reports an issue with the container-executor #7290

Conversation

brumi1024 commented Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Jan 15, 2025

Uh oh!

hadoop-yetus commented Jan 16, 2025

Uh oh!

p-szucs commented Jan 16, 2025

Uh oh!

slfan1989 commented Jan 17, 2025

Uh oh!

brumi1024 commented Jan 20, 2025

Uh oh!

hadoop-yetus commented Jan 20, 2025

Uh oh!

Uh oh!

Uh oh!

brumi1024 commented Jan 15, 2025 •

edited

Loading