[SPARK-28097][SQL] Map ByteType to SMALLINT for PostgresDialect

[mojodna]

What changes were proposed in this pull request?

PostgreSQL doesn't have TINYINT, which would map directly, but SMALLINTs are sufficient for uni-directional translation.

A side-effect of this fix is that AggregatedDialect is now usable with multiple dialects targeting jdbc:postgresql, as PostgresDialect.getJDBCType no longer throws (for which reason backporting this fix would be lovely):

spark/sql/core/src/main/scala/org/apache/spark/sql/jdbc/AggregatedDialect.scala

Line 42 in 1217996

dialects.flatMap(_.getJDBCType(dt)).headOption

dialects.flatMap currently throws on the first attempt to get a JDBC type preventing subsequent dialects in the chain from providing an alternative.

How was this patch tested?

Unit tests.

[dongjoon-hyun]

Thank you for making a PR, @mojodna .

1. In general, PostgreSQL users will not use unsupported types.
   I'm wondering if your goal is the one you mentioned as side-effects AggregatedDialect is now usable. Could you describe a little bit more about the use cases?
2. Also, please create a JIRA issue for this suggestion and use the JIRA id to the PR title. PR is valuable, but JIRA issue also becomes a history.

[srowen]

File a JIRA, yes. You've checked this works on postgres?

[mojodna]

@srowen yes, it works on Postgres.

@dongjoon-hyun https://issues.apache.org/jira/browse/SPARK-28100 describes the underlying problem (i.e. why I can't provide a custom dialect with an alternate mapping).

The actual use-case is writing spatial data from Spark (using Spark JTS) to Postgres using the JDBC sink (where I may need to map sqlType == Types.OTHER && typeName == "geometry" to `BinaryType, but that's entirely separate (and can be handled by my own dialect).

ByteTypes are in my data model as type discriminators, which is how I stumbled on this.

Some time later, I realized that I can cast my bytes to shorts before handing off to the JDBC sink, so this is more a case of things not working as I'd expect them to.

[gatorsmile]

Add a test to PostgresIntegrationSuite?

[gatorsmile]

ok to test

[mojodna]

@gatorsmile PostgresIntegrationSuite seems to just contain read-related tests (JDBC → DF); JDBCSuite.scala tests the mappings when writing (DF → JDBC).

[SparkQA]

Test build #106641 has finished for PR 24845 at commit 50cb99a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

@gatorsmile given the comment at #24845 (comment) are you OK with this change?

[gatorsmile]

@mojodna @srowen I think we still needs an end-to-end test to ensure it works well in postgreSQL. Is it hard to do it in PostgresIntegrationSuite ?

[srowen]

@mojodna what do you think about adding a simple additional test here to verify?

[mojodna]

I'm swamped for the next couple weeks, but sure thing. Is there a specific test within PostgresIntegrationSuite that I should use as a template?

[maropu]

PostgresIntegrationSuite can run (DF → JDBC) tests easily like this;

  test("write byte as smallint") {
    sqlContext.createDataFrame(Seq((1.toByte, 2.toShort)))
      .write.jdbc(jdbcUrl, "byte_to_smallint_test", new Properties)
    val df = sqlContext.read.jdbc(jdbcUrl, "byte_to_smallint_test", new Properties)
    val schema = df.schema
    assert(schema(0).dataType == ShortType)
    assert(schema(1).dataType == ShortType)
    val rows = df.collect()
    assert(rows.length === 1)
    assert(rows(0).getShort(0) === 1)
    assert(rows(0).getShort(1) === 2)
  }

[dongjoon-hyun]

Gentle ping, @mojodna .

[mojodna]

Thanks @dongjoon-hyun. We just moved and are getting settled in, so sometime next week looks very likely.

[dongjoon-hyun]

Thank you, @mojodna .

[mojodna]

Updated. Thanks @maropu for doing the hard part!

[SparkQA]

Test build #107798 has finished for PR 24845 at commit 67713ee.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. Merged to master.
Thank you, @mojodna , @srowen , @gatorsmile , @maropu !

[dongjoon-hyun]

Thank you so much for your contribution, @mojodna .
You are added to the Apache Spark contributor group and SPARK-28097 is assigned to you.