[SPARK-35085][SQL] Get columns operation should handle ANSI interval column properly #32345

beliefer · 2021-04-26T06:30:45Z

What changes were proposed in this pull request?

This PR let JDBC clients identify ANSI interval columns properly.

Why are the changes needed?

This PR is similar to #29539.
JDBC users can query interval values through thrift server, create views with ansi interval columns, e.g.
CREATE global temp view view1 as select interval '1-1' year to month as I;
but when they want to get the details of the columns of view1, the will fail with Unrecognized type name: YEAR-MONTH INTERVAL

Caused by: java.lang.IllegalArgumentException: Unrecognized type name: YEAR-MONTH INTERVAL
	at org.apache.spark.sql.hive.thriftserver.SparkGetColumnsOperation.toJavaSQLType(SparkGetColumnsOperation.scala:190)
	at org.apache.spark.sql.hive.thriftserver.SparkGetColumnsOperation.$anonfun$addToRowSet$1(SparkGetColumnsOperation.scala:206)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.sql.hive.thriftserver.SparkGetColumnsOperation.addToRowSet(SparkGetColumnsOperation.scala:198)
	at org.apache.spark.sql.hive.thriftserver.SparkGetColumnsOperation.$anonfun$runInternal$7(SparkGetColumnsOperation.scala:109)
	at org.apache.spark.sql.hive.thriftserver.SparkGetColumnsOperation.$anonfun$runInternal$7$adapted(SparkGetColumnsOperation.scala:109)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.sql.hive.thriftserver.SparkGetColumnsOperation.$anonfun$runInternal$5(SparkGetColumnsOperation.scala:109)
	at org.apache.spark.sql.hive.thriftserver.SparkGetColumnsOperation.$anonfun$runInternal$5$adapted(SparkGetColumnsOperation.scala:107)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.sql.hive.thriftserver.SparkGetColumnsOperation.runInternal(SparkGetColumnsOperation.scala:107)
	... 34 more

Does this PR introduce any user-facing change?

Yes. Let hive JDBC recognize ANSI interval.

How was this patch tested?

Jenkins test.

SparkQA · 2021-04-26T07:07:07Z

Test build #137936 has finished for PR 32345 at commit 23c8cf2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-26T07:21:42Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42458/

SparkQA · 2021-04-26T08:08:39Z

Test build #137939 has finished for PR 32345 at commit 211dbf2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-26T08:33:55Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42461/

beliefer · 2021-04-26T09:45:37Z

ping @MaxGekk

MaxGekk

@beliefer Could you fill in the PR description, please.

beliefer · 2021-04-26T10:16:01Z

@beliefer Could you fill in the PR description, please.

I'm sorry! I forgot it.

MaxGekk · 2021-04-26T12:28:10Z

...tserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetColumnsOperation.scala

+    case IntegerType | YearMonthIntervalType => java.sql.Types.INTEGER
+    case LongType | DayTimeIntervalType => java.sql.Types.BIGINT


hmm, I am not sure that we should expose ANSI intervals as raw integers/longs via JDBC. I would consider strings (preferable) or java.time.Duration/Period. @cloud-fan @srielau WDYT?

Good question! I have the confusion too.
cc @cloud-fan

This is the metadata, where do we handle the data? e.g. if we want to return string or Duration/Period, where shall we instantiate string or Duration/Period values?

See the PR #32121. We should return strings. Since there is no appropriate type in java.sql.* for intervals. I believe we should return the same as for CalendarIntervalType - java.sql.Types.OTHER.

@beliefer Could you handle YearMonthIntervalType and DayTimeIntervalType separately, and return java.sql.Types.OTHER

MaxGekk · 2021-04-27T14:34:57Z

...tserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetColumnsOperation.scala

  private def getColumnSize(typ: DataType): Option[Int] = typ match {
    case dt @ (BooleanType | _: NumericType | DateType | TimestampType |
-               CalendarIntervalType | NullType) =>
+               CalendarIntervalType | NullType | YearMonthIntervalType | DayTimeIntervalType) =>


Size of what does it return? CalendarIntervalType, YearMonthIntervalType, DayTimeIntervalType are returned as strings in rowSets.

CalendarIntervalType return it's defaultSize (4 + 4 + 8 = 16).
It seems YearMonthIntervalType and DayTimeIntervalType should return defaultSize too.

SparkQA · 2021-04-28T05:08:10Z

Test build #138023 has finished for PR 32345 at commit 4c594bb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-28T05:13:19Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42542/

MaxGekk · 2021-04-28T05:58:06Z

+1, LGTM. Merging to master.
Thank you, @beliefer and @cloud-fan for review.

beliefer · 2021-04-28T06:42:44Z

@MaxGekk @cloud-fan Thank you.

HyukjinKwon

LGTM2

Get columns operation should handle ANSI interval column properly

23c8cf2

github-actions bot added the SQL label Apr 26, 2021

test

211dbf2

beliefer changed the title ~~[WIP][SPARK-35085][SQL] Get columns operation should handle ANSI interval column properly~~ [SPARK-35085][SQL] Get columns operation should handle ANSI interval column properly Apr 26, 2021

MaxGekk reviewed Apr 26, 2021

View reviewed changes

MaxGekk reviewed Apr 27, 2021

View reviewed changes

Update code

4c594bb

MaxGekk approved these changes Apr 28, 2021

View reviewed changes

MaxGekk closed this in 56bb815 Apr 28, 2021

HyukjinKwon reviewed Apr 29, 2021

View reviewed changes

		case IntegerType \| YearMonthIntervalType => java.sql.Types.INTEGER
		case LongType \| DayTimeIntervalType => java.sql.Types.BIGINT

[SPARK-35085][SQL] Get columns operation should handle ANSI interval column properly #32345

[SPARK-35085][SQL] Get columns operation should handle ANSI interval column properly #32345

Uh oh!

Conversation

beliefer commented Apr 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Apr 26, 2021

Uh oh!

SparkQA commented Apr 26, 2021

Uh oh!

SparkQA commented Apr 26, 2021

Uh oh!

SparkQA commented Apr 26, 2021

Uh oh!

beliefer commented Apr 26, 2021

Uh oh!

MaxGekk left a comment

Choose a reason for hiding this comment

Uh oh!

beliefer commented Apr 26, 2021

Uh oh!

MaxGekk Apr 26, 2021

Choose a reason for hiding this comment

Uh oh!

beliefer Apr 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan Apr 27, 2021

Choose a reason for hiding this comment

Uh oh!

MaxGekk Apr 27, 2021

Choose a reason for hiding this comment

Uh oh!

beliefer Apr 28, 2021

Choose a reason for hiding this comment

Uh oh!

MaxGekk Apr 27, 2021

Choose a reason for hiding this comment

Uh oh!

beliefer Apr 28, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 28, 2021

Uh oh!

SparkQA commented Apr 28, 2021

Uh oh!

MaxGekk commented Apr 28, 2021

Uh oh!

beliefer commented Apr 28, 2021

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

beliefer commented Apr 26, 2021 •

edited

Loading

beliefer Apr 26, 2021 •

edited

Loading