Skip to content

Commit dde0d2f

Browse files
committed
[SPARK-29991][INFRA] Support test-hive1.2 in PR Builder
### What changes were proposed in this pull request? Currently, Apache Spark PR Builder using `hive-1.2` for `hadoop-2.7` and `hive-2.3` for `hadoop-3.2`. This PR aims to support `[test-hive1.2]` in PR Builder in order to cut the correlation between `hive-1.2/2.3` to `hadoop-2.7/3.2`. After this PR, the PR Builder will use `hive-2.3` by default for all profiles (if there is no `test-hive1.2`.) ### Why are the changes needed? This new tag allows us more flexibility. ### Does this PR introduce any user-facing change? No. (This is a dev-only change.) ### How was this patch tested? Check the Jenkins triggers in this PR. **BEFORE** ``` ======================================================================== Building Spark ======================================================================== [info] Building Spark using SBT with these arguments: -Phadoop-2.7 -Phive-1.2 -Pyarn -Pkubernetes -Phive -Phadoop-cloud -Pspark-ganglia-lgpl -Phive-thriftserver -Pkinesis-asl -Pmesos test:package streaming-kinesis-asl-assembly/assembly ``` **AFTER** 1. Title: [[SPARK-29991][INFRA][test-hive1.2] Support `test-hive1.2` in PR Builder](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114550/testReport) ``` ======================================================================== Building Spark ======================================================================== [info] Building Spark using SBT with these arguments: -Phadoop-2.7 -Phive-1.2 -Pkinesis-asl -Phadoop-cloud -Pyarn -Phive -Pmesos -Pspark-ganglia-lgpl -Pkubernetes -Phive-thriftserver test:package streaming-kinesis-asl-assembly/assembly ``` 2. Title: [[SPARK-29991][INFRA] Support `test hive1.2` in PR Builder](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114551/testReport) - Note that I removed the hyphen intentionally from `test-hive1.2`. ``` ======================================================================== Building Spark ======================================================================== [info] Building Spark using SBT with these arguments: -Phadoop-2.7 -Phive-thriftserver -Pkubernetes -Pspark-ganglia-lgpl -Phadoop-cloud -Phive -Pmesos -Pyarn -Pkinesis-asl test:package streaming-kinesis-asl-assembly/assembly ``` Closes #26695 from dongjoon-hyun/SPARK-29991. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 9459833 commit dde0d2f

File tree

1 file changed

+7
-3
lines changed

1 file changed

+7
-3
lines changed

dev/run-tests.py

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -283,12 +283,16 @@ def get_hadoop_profiles(hadoop_version):
283283
"""
284284

285285
sbt_maven_hadoop_profiles = {
286-
"hadoop2.7": ["-Phadoop-2.7", "-Phive-1.2"],
287-
"hadoop3.2": ["-Phadoop-3.2", "-Phive-2.3"],
286+
"hadoop2.7": ["-Phadoop-2.7"],
287+
"hadoop3.2": ["-Phadoop-3.2"],
288288
}
289289

290290
if hadoop_version in sbt_maven_hadoop_profiles:
291-
return sbt_maven_hadoop_profiles[hadoop_version]
291+
if ("ghprbPullTitle" in os.environ and
292+
"test-hive1.2" in os.environ["ghprbPullTitle"].lower()):
293+
return sbt_maven_hadoop_profiles[hadoop_version] + ["-Phive-1.2"]
294+
else:
295+
return sbt_maven_hadoop_profiles[hadoop_version]
292296
else:
293297
print("[error] Could not find", hadoop_version, "in the list. Valid options",
294298
" are", sbt_maven_hadoop_profiles.keys())

0 commit comments

Comments
 (0)