Spark: Cannot create expression literals from Instant

[RussellSpitzer]

Summary:
Literals.from doesn't have support for creating literals from java.time.Instant which can be passed in Spark 3.1.1

Cannot create expression literal from java.time.Instant: 2021-04-25T19:00:00Z
at org.apache.iceberg.expressions.Literals.from(Literals.java:85)
at org.apache.iceberg.expressions.UnboundPredicate.(UnboundPredicate.java:40)

---

Long story

Reports from some users testing out Spark 3.1 which has an extremely weird behavior.

1. Create a table with a timestamp partition key

spark.sql("CREATE TABLE foo (ts timestamp) USING ICEBERG PARTITIONED BY (hours(ts)) LOCATION '/Users/russellspitzer/temp'")
spark.sql("INSERT into foo VALUES (cast(\"2021-04-25T19:01:00+00:00\" as timestamp))")

2. Query this table

SELECT * from foo WHERE ts >= \"2021-04-25T19:00:00+00:00\" and ts < \"2021-04-25T20:00:00+00:00\"

The user reported the exception I noted at the top of this issue, I attempt to reproduce and cannot. I then find the following matrix of behaviors.

Spark 3.0.2 - spark-shell - No Issue
Spark 3.0.2 - spark-sql - No issue
Spark 3.1.1 - spark-shell - No issue
Spark 3.1.1 - spark-sql - Exception - Query Fails

While i'm slightly confused as why there is a difference in parsing behavior between spark-shell and spark-sql in 3.1.1 I think the fix is pretty clear and we need to just add support for creating timestamp literals from java.time.Instants

[RussellSpitzer]

https://issues.apache.org/jira/browse/SPARK-26902 Makes me think we will probably have to support local date as well

[Poncirus]

Is this the same issue? https://issues.apache.org/jira/browse/SPARK-33867

[RussellSpitzer]

Looks like the same core issue but for the JDBC Connector/Dialect code

…

On Thu, May 20, 2021 at 6:34 AM Liao Hanwen ***@***.***> wrote: Is this the same issue? https://issues.apache.org/jira/browse/SPARK-33867 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2530 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADE2YP54TYN3DOV4MNZPKDTOTXUZANCNFSM43VHGUHQ> .

[Poncirus]

I may find the reason.
See https://issues.apache.org/jira/browse/SPARK-31910 and apache/spark#28729. Configure spark.sql.datetime.java8API.enabled is set to true in thrift SQL CLI, whose default setting is false.
I work around this issue by adding "set spark.sql.datetime.java8API.enabled=false;" in my spark SQL temporarily. Looking forward to new release of iceberg.

[RussellSpitzer]

Ah ha! Thanks for tracking that down! I knew there had to be something floating around that caused the issue

…

Sent from my iPhone

On May 21, 2021, at 4:45 AM, Liao Hanwen ***@***.***> wrote:  I may find the reason. See https://issues.apache.org/jira/browse/SPARK-31910 and apache/spark#28729. Configure spark.sql.datetime.java8API.enabled is set to true in thrift SQL CLI, whose default setting is false. I work around this issue by adding "set spark.sql.datetime.java8API.enabled=false;" in my spark SQL temporarily. Looking forward to new release of iceberg. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.