[SPARK-27737][SQL] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile #24620

wangyum · 2019-05-15T22:56:22Z

What changes were proposed in this pull request?

This PR aims to upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile.

Release Notes - Hive - Version 2.3.5

[HIVE-21536] - Backport HIVE-17764 to branch-2.3
[HIVE-21585] - Upgrade branch-2.3 to ORC 1.3.4
[HIVE-21639] - Spark test failed since HIVE-10632
[HIVE-21680] - Backport HIVE-17644 to branch-2 and branch-2.3

https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345394&styleName=Text&projectId=12310843

How was this patch tested?

This PR is tested in two ways.

Pass the Jenkins with the default configuration for Hive Metastore Client testing.
Pass the Jenkins with test-hadoop3.2 configuration for Hadoop 3.2 testing.

SparkQA · 2019-05-15T23:52:21Z

Test build #105431 has finished for PR 24620 at commit 34e4817.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-05-16T09:52:09Z

retest this please

HyukjinKwon

yes, looks good if the tests pass.

wangyum · 2019-05-16T09:59:18Z

The Hive 2.3.5 jars is not in the maven repository. May be need to wait for a while?

SparkQA · 2019-05-16T10:00:20Z

Test build #105451 has started for PR 24620 at commit 34e4817.

HyukjinKwon · 2019-05-16T10:14:05Z

okie.

dongjoon-hyun

@wangyum . Please merge this and #24621 because we had better update the dependency file consistently across all profiles from Hive 2.3.5.

cc @gatorsmile and @srowen .

dongjoon-hyun · 2019-05-16T16:11:12Z

BTW, @wangyum . To get reviews, you had better make a PR after the maven repository is ready next time. :)

HyukjinKwon · 2019-05-16T23:35:57Z

Yea agree.

wangyum · 2019-05-17T00:58:21Z

retest this please

SparkQA · 2019-05-17T02:44:40Z

Test build #105478 has finished for PR 24620 at commit 34e4817.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-05-20T07:05:02Z

Test build #105556 has finished for PR 24620 at commit 4fd1f83.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-05-20T07:05:38Z

retest this please

SparkQA · 2019-05-20T09:20:58Z

Test build #105558 has finished for PR 24620 at commit 4fd1f83.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-05-20T16:34:42Z

Thank you for updating, @wangyum . After #24639 is merged, I'll retrigger this with [test-hadoop3.2][test-maven].

dongjoon-hyun · 2019-05-20T20:05:20Z

Retest this please.

dongjoon-hyun · 2019-05-20T20:21:00Z

I updated the PR title and description. The last Jenkins run is running with the following.

[info] Building Spark using Maven with these arguments: 
-Phadoop-3.2 -Pkubernetes -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl
-Phive -Pmesos clean package -DskipTests

wangyum · 2019-05-20T21:45:40Z

Hive 2.3.5 should support test with sbt.

wangyum · 2019-05-20T21:46:11Z

retest this please

dongjoon-hyun · 2019-05-20T22:22:28Z

Thank you for additional one, @wangyum . To sum up,

sbt on hadoop-2.7 was tested via [SPARK-27737][SQL] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile #24620 (comment)

- [info] Building Spark using SBT with these arguments:  -Phadoop-2.7 -Pkubernetes -Phive-thriftserver -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos test:package streaming-kinesis-asl-assembly/assembly

sbt on hadoop-3.2 is running on Test build #105579 has started
maven on hadoop-3.2 is running on Test build #105578 has started

@wangyum . Do you want to trigger maven on hadoop-2.7, too?

dongjoon-hyun · 2019-05-21T00:58:44Z

Yep. So, if you want, you can trigger it to verify this PR completely.

Do you think we need test maven on hadoop-2.7?

wangyum · 2019-05-21T01:05:03Z

retest this please

dongjoon-hyun · 2019-05-21T04:46:18Z

Ur, @wangyum . During merging, it seems that we missed to upgrade hadoop-3.2 profile properly.

      <id>hadoop-3.2</id>
        ...
        <hive.version.short>2.3.4</hive.version.short>

Please update hadoop-3.2 profile and update the manifests.

wangyum · 2019-05-21T04:55:13Z

How about change hive.version.short to 2.3.0?

SparkQA · 2019-05-21T05:49:06Z

Test build #105585 has finished for PR 24620 at commit 4fd1f83.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-05-21T05:59:31Z

Is there a reason to use that?

How about change hive.version.short to 2.3.0?

SparkQA · 2019-05-21T06:18:53Z

Test build #105587 has finished for PR 24620 at commit 4fd1f83.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-05-21T06:23:24Z

2 reasons:

Avoid changing these directories: sql/core/v2.3.4/src, sql/hive-thriftserver/v2.3.4 and hive.version.short if we upgrade Hive 2.3 to 2.3.6, 2.3.7 ...
2.3.0 seems more natural:

dongjoon-hyun · 2019-05-21T06:41:40Z

Does something fail with 2.3.5? Before Spark 3.0.0 release, Hive 2.3.5 seems to be the last and the most stable Hive release. So, shall we catch up at this time? It would be great if you do the following.

dev/test-dependencies.sh --replace-manifest
sql/core/v2.3.4/src -> sql/core/v2.3.5/src

For shortVersion, we use 1.2.1 previously. So, 2.3.5 looks fine to me.

SparkQA · 2019-05-21T07:27:37Z

Test build #105605 has finished for PR 24620 at commit 0f2eb07.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

wangyum · 2019-05-21T07:56:20Z

@HyukjinKwon It add -Phive-thriftserver after merge upstream:

========================================================================
Building Spark
========================================================================
[info] Building Spark using SBT with these arguments:  -Phadoop-3.2 -Pkubernetes -Phive-thriftserver -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos test:package streaming-kinesis-asl-assembly/assembly

SparkQA · 2019-05-21T09:31:53Z

Test build #105607 has finished for PR 24620 at commit 32fc1ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-05-21T14:18:12Z

========================================================================
Building Spark
========================================================================
[info] Building Spark using SBT with these arguments:  -Phadoop-3.2 -Pkubernetes -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos test:package streaming-kinesis-asl-assembly/assembly

Seems not. which build are you referring?

HyukjinKwon · 2019-05-21T14:19:25Z

This build https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105605/ was against 0f2eb07 before merging upstream.

wangyum · 2019-05-21T14:25:11Z

Retest this please

HyukjinKwon · 2019-05-21T14:36:34Z

retest this please

SparkQA · 2019-05-21T16:52:49Z

Test build #105626 has finished for PR 24620 at commit 32fc1ca.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-05-21T17:07:09Z

retest this please

SparkQA · 2019-05-21T19:13:39Z

Test build #105631 has finished for PR 24620 at commit 32fc1ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-05-21T19:54:26Z

Retest this please.

SparkQA · 2019-05-22T01:04:29Z

Test build #105636 has finished for PR 24620 at commit 32fc1ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-05-22T01:23:18Z

Merged to master.

Upgrade to 2.3.5 for Hive Metastore Client 2.3

34e4817

wangyum mentioned this pull request May 15, 2019

[WIP][SPARK-27738][BUILD][test-hadoop3.2] Upgrade the built-in Hive to 2.3.5 for hadoop-3.2 #24621

Closed

felixcheung approved these changes May 16, 2019

View reviewed changes

HyukjinKwon approved these changes May 16, 2019

View reviewed changes

dongjoon-hyun requested changes May 16, 2019

View reviewed changes

srowen approved these changes May 17, 2019

View reviewed changes

Upgrade built-in Hive to 2.3.5

4fd1f83

HyukjinKwon approved these changes May 20, 2019

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-27737][SQL] Upgrade to 2.3.5 for Hive Metastore Client 2.3~~ [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to 2.3.5 for Hive Metastore Client 2.3 May 20, 2019

dongjoon-hyun changed the title ~~[SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to 2.3.5 for Hive Metastore Client 2.3~~ [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 May 20, 2019

dongjoon-hyun changed the title ~~[SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5~~ [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 20, 2019

wangyum changed the title ~~[SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile~~ [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 20, 2019

wangyum changed the title ~~[SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile~~ [SPARK-27737][SQL][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 21, 2019

wangyum changed the title ~~[SPARK-27737][SQL][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile~~ [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 21, 2019

wangyum added 2 commits May 21, 2019 15:16

Change hive.version.short to 2.3.5

0f2eb07

Merge remote-tracking branch 'upstream/master' into SPARK-27737

32fc1ca

dongjoon-hyun changed the title ~~[SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile~~ [SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 21, 2019

HyukjinKwon changed the title ~~[SPARK-27737][SQL][test-hadoop3.2][test-maven] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile~~ [SPARK-27737][SQL] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile May 22, 2019

HyukjinKwon closed this in 6cd1efd May 22, 2019

[SPARK-27737][SQL] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile #24620

[SPARK-27737][SQL] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile #24620

Uh oh!

Conversation

wangyum commented May 15, 2019 • edited by dongjoon-hyun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented May 15, 2019

Uh oh!

HyukjinKwon commented May 16, 2019

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

wangyum commented May 16, 2019

Uh oh!

SparkQA commented May 16, 2019

Uh oh!

HyukjinKwon commented May 16, 2019

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 16, 2019

Uh oh!

HyukjinKwon commented May 16, 2019

Uh oh!

wangyum commented May 17, 2019

Uh oh!

SparkQA commented May 17, 2019

Uh oh!

SparkQA commented May 20, 2019

Uh oh!

wangyum commented May 20, 2019

Uh oh!

SparkQA commented May 20, 2019

Uh oh!

dongjoon-hyun commented May 20, 2019

Uh oh!

dongjoon-hyun commented May 20, 2019

Uh oh!

dongjoon-hyun commented May 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangyum commented May 20, 2019

Uh oh!

wangyum commented May 20, 2019

Uh oh!

dongjoon-hyun commented May 20, 2019

Uh oh!

dongjoon-hyun commented May 21, 2019

Uh oh!

wangyum commented May 21, 2019

Uh oh!

dongjoon-hyun commented May 21, 2019

Uh oh!

wangyum commented May 21, 2019

Uh oh!

SparkQA commented May 21, 2019

Uh oh!

dongjoon-hyun commented May 21, 2019

Uh oh!

SparkQA commented May 21, 2019

Uh oh!

wangyum commented May 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented May 21, 2019

Uh oh!

SparkQA commented May 21, 2019

Uh oh!

wangyum commented May 21, 2019

Uh oh!

SparkQA commented May 21, 2019

Uh oh!

HyukjinKwon commented May 21, 2019

Uh oh!

HyukjinKwon commented May 21, 2019

wangyum commented May 15, 2019 •

edited by dongjoon-hyun

Loading

dongjoon-hyun commented May 20, 2019 •

edited

Loading

wangyum commented May 21, 2019 •

edited

Loading