MAPREDUCE-7523. MapReduce Task-Level Security Enforcement #8100
+498
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
K0K0V0K
force-pushed
the
MAPREDUCE-7523
branch
from
6705730 to
3866c6e
Compare
The goal of this feature tp provide a configurable mechanism to control which users are allowed to execute specific MapReduce jobs. This feature aims to prevent unauthorized or potentially harmful mapper/reducer implementations from running within the Hadoop cluster. In the standard Hadoop MapReduce execution flow: 1) A MapReduce job is submitted by a user. 2) The job is registered with the Resource Manager (RM). 3) The RM assigns the job to a Node Manager (NM), where the Application Master (AM) for the job is launched. 4) The AM requests additional containers from the cluster, to be able to start tasks. 5) The NM launches those containers, and the containers execute the mapper/reducer tasks defined by the job. The proposed feature introduces a security filtering mechanism inside the Application Master. Before mapper or reducer tasks are launched, the AM will verify that the user-submitted MapReduce code complies with a cluster-defined security policy. This ensures that only approved classes or packages can be executed inside the containers. The goal is to protect the cluster from unwanted or unsafe task implementations, such as custom code that may introduce performance, stability, or security risks. Upon receiving job metadata, the Application Master will: 1) Check the feature is enabled. 2) Check the user who submitted the job is allowed to bypass the security check. 3) Compare classes in job config against the denied task list. 4) If job is not authorised an exception will be thrown and AM will fail. New Configs Enables MapReduce Task-Level Security Enforcement When enabled, the Application Master performs validation of user-submitted mapper, reducer, and other task-related classes before launching containers. This mechanism protects the cluster from running disallowed or unsafe task implementations as defined by administrator-controlled policies. - Property name: mapreduce.security.enabled - Property type: boolean - Default: false (security disabled) MapReduce Task-Level Security Enforcement: Property Domain Defines the set of MapReduce configuration keys that represent user-supplied class names involved in task execution (e.g., mapper, reducer, partitioner). The Application Master examines the values of these properties and checks whether any referenced class is listed in denied tasks. Administrators may override this list to expand or restrict the validation domain. - Property name: mapreduce.security.property-domain - Property type: list of configuration keys - Default: map.sort.class mapreduce.job.classloader.system.classes mapreduce.job.combine.class mapreduce.job.combiner.group.comparator.class mapreduce.job.end-notification.custom-notifier-class mapreduce.job.inputformat.class mapreduce.job.map.class mapreduce.job.map.output.collector.class mapreduce.job.output.group.comparator.class mapreduce.job.output.key.class mapreduce.job.output.key.comparator.class mapreduce.job.output.value.class mapreduce.job.outputformat.class mapreduce.job.partitioner.class mapreduce.job.reduce.class mapreduce.map.output.key.class mapreduce.map.output.value.class MapReduce Task-Level Security Enforcement: Denied Tasks Specifies the list of disallowed task implementation classes or packages. If a user submits a job whose mapper, reducer, or other task-related classes match any entry in this blacklist. - Property name: mapreduce.security.denied-tasks - Property type: list of class name or package patterns - Default: empty - Example: org.apache.hadoop.streaming,org.apache.hadoop.examples.QuasiMonteCarlo MapReduce Task-Level Security Enforcement: Allowed Users Specifies users who may bypass the blacklist defined in denied tasks. This whitelist is intended for trusted or system-level workflows that may legitimately require the use of restricted task implementations. If the submitting user is listed here, blacklist enforcement is skipped, although standard Hadoop authentication and ACL checks still apply. - Property name: mapreduce.security.allowed-users - Property type: list of usernames - Default: empty - Example: alice,bob
K0K0V0K
force-pushed
the
MAPREDUCE-7523
branch
from
3866c6e to
8c688bb
Compare
K0K0V0K
force-pushed
the
MAPREDUCE-7523
branch
from
018e4cb to
40457a3
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
...ava/org/apache/hadoop/mapreduce/v2/app/security/authorize/TestTaskLevelSecurityEnforcer.java
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
The goal of this feature tp provide a configurable mechanism to control which users are allowed to execute specific MapReduce jobs. This feature aims to prevent unauthorized or potentially harmful mapper/reducer implementations from running within the Hadoop cluster.
In the standard Hadoop MapReduce execution flow:
The proposed feature introduces a security filtering mechanism inside the Application Master. Before mapper or reducer tasks are launched, the AM will verify that the user-submitted MapReduce code complies with a cluster-defined security policy. This ensures that only approved classes or packages can be executed inside the containers. The goal is to protect the cluster from unwanted or unsafe task implementations, such as custom code that may introduce performance, stability, or security risks.
Upon receiving job metadata, the Application Master will:
New Configs
Enables MapReduce Task-Level Security Enforcement
When enabled, the Application Master performs validation of user-submitted mapper, reducer, and other task-related classes before launching containers. This mechanism protects the cluster from running disallowed or unsafe task implementations as defined by administrator-controlled policies.
MapReduce Task-Level Security Enforcement: Property Domain
Defines the set of MapReduce configuration keys that represent user-supplied class names involved in task execution (e.g., mapper, reducer, partitioner). The Application Master examines the values of these properties and checks whether any referenced class is listed in denied tasks. Administrators may override this list to expand or restrict the validation domain.
mapreduce.job.combine.class
mapreduce.job.combiner.group.comparator.class
mapreduce.job.end-notification.custom-notifier-class
mapreduce.job.inputformat.class
mapreduce.job.map.class
mapreduce.job.map.output.collector.class
mapreduce.job.output.group.comparator.class
mapreduce.job.output.key.class
mapreduce.job.output.key.comparator.class
mapreduce.job.output.value.class
mapreduce.job.outputformat.class
mapreduce.job.partitioner.class
mapreduce.job.reduce.class
mapreduce.map.output.key.class
mapreduce.map.output.value.class
MapReduce Task-Level Security Enforcement: Denied Tasks
Specifies the list of disallowed task implementation classes or packages. If a user submits a job whose mapper, reducer, or other task-related classes match any entry in this blacklist.
MapReduce Task-Level Security Enforcement: Allowed Users
Specifies users who may bypass the blacklist defined in denied tasks. This whitelist is intended for trusted or system-level workflows that may legitimately require the use of restricted task implementations. If the submitting user is listed here, blacklist enforcement is skipped, although standard Hadoop authentication and ACL checks still apply.
How was this patch tested?
UT was run
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?