[SPARK-28034][SQL][TEST] Port with.sql #24860

peter-toth · 2019-06-13T09:52:05Z

What changes were proposed in this pull request?

This PR is to port with.sql from PostgreSQL regression tests. https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/with.sql

The expected results can be found in the link: https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/expected/with.out

When porting the test cases, found 7 PostgreSQL specific features that do not exist in Spark SQL:

SPARK-19799 Support WITH clause in subqueries
SPARK-24497 Support recursive SQL query
SPARK-28297 Handling outer links in CTE subquery expressions
SPARK-28296 Improved VALUES support
SPARK-28146 Support IS OF type predicate
SPARK-28147 Support RETURNING clause
SPARK-27878 Support ARRAY(sub-SELECT) expressions

Also, found one inconsistent behavior:

SPARK-28299 Evaluation of multiple CTE uses

Also, added the following notes:

Spark SQL doesn't support DELETE statement
Spark SQL doesn't support UPDATE statement
Spark SQL doesn't support RULEs
Spark SQL doesn't support UNIQUE constraints
Spark SQL doesn't support ON CONFLICT clause
Spark SQL doesn't support TRIGGERs
Spark SQL doesn't support INHERITS clause

How was this patch tested?

N/A

peter-toth · 2019-06-13T10:01:12Z

This is a work in progress PR as currently almost nothing can be ported and most of the with.sql is commented out, but I believe SPARK-19799, SPARK-28002 and SPARK-24497 could help a lot.
(Actually, SPARK-24497 could be moved under SPARK-27764 as a subtask.)

I left a lot of TODOs in the PR and will try to clean them up next week.

sql/core/src/test/resources/sql-tests/inputs/pgSQL/with.sql

peter-toth · 2019-06-25T14:27:51Z

@dongjoon-hyun, @gatorsmile, @wangyum I finished the analysis of with.sql. Unfortunately there is a large discrepancy between PostgreSQL and Spark SQL so most of the file is still commented out.

I have a PR open to address WITH in subqueries: #24831,
and another work-in-progress one to support recursive queries: #23531 that probably could help a lot.

dongjoon-hyun · 2019-06-26T16:39:58Z

ok to test

SparkQA · 2019-06-26T20:05:28Z

Test build #106939 has finished for PR 24860 at commit fd8abe0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/test/resources/sql-tests/inputs/pgSQL/with.sql

SparkQA · 2019-06-28T16:56:19Z

Test build #107009 has finished for PR 24860 at commit b9e3388.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-07-04T02:53:05Z

Retest this please.

sql/core/src/test/resources/sql-tests/results/pgSQL/with.sql.out

sql/core/src/test/resources/sql-tests/inputs/pgSQL/with.sql

SparkQA · 2019-07-04T06:00:48Z

Test build #107214 has finished for PR 24860 at commit b9e3388.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-07-04T16:43:10Z

Test build #107235 has finished for PR 24860 at commit 292be80.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-07-04T19:39:52Z

Since #25054 is merged, could you regenerate the result once more?

peter-toth · 2019-07-05T09:04:06Z

Since #25054 is merged, could you regenerate the result once more?

Sure, done.

SparkQA · 2019-07-05T11:48:51Z

Test build #107271 has finished for PR 24860 at commit ad7a86c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

peter-toth · 2019-07-08T13:15:32Z

I still left 4 TODOs in the file, could you please help me @dongjoon-hyun what to do with these?

I can't find an option in Spark SQL to query the DDL of a view, but I might be missing something. Do you know if there is an option to do it? If there isn't, shall I create a ticket for it?
WITH is a reserved keyword in PostgreSQL, but it isn't in Spark SQL. How shall we handle this?

Also please check the notes in the PR description. I left these notes in the ported SQL file where I though it makes no sense to create a ticket.

sql/core/src/test/resources/sql-tests/inputs/pgSQL/with.sql

dongjoon-hyun · 2019-07-08T14:59:30Z

Spark had Views-to-DDL before, but it's removed. IIRC, there is a ticket. If you search, you can find the deletion ticket.

SparkQA · 2019-07-08T15:35:33Z

Test build #107345 has finished for PR 24860 at commit 5b9997e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

peter-toth · 2019-07-08T15:41:09Z

Views-to-DDL

Spark had Views-to-DDL before, but it's removed. IIRC, there is a ticket. If you search, you can find the deletion ticket.

Oh, it seems SHOW CREATE TABLE does work on views.

SparkQA · 2019-07-08T15:59:57Z

Test build #107351 has finished for PR 24860 at commit 4a356e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-07-10T17:30:08Z

Retest this please.

dongjoon-hyun · 2019-07-10T17:32:19Z

So, the remaining things looks like cleaning up the TODOs. Did I understand correctly?

peter-toth · 2019-07-10T19:08:51Z

So, the remaining things looks like cleaning up the TODOs. Did I understand correctly?

Yes, and actually the only remaining question is if WITH should be a reserved keyword as it is in PostgreSQL?

dongjoon-hyun · 2019-07-10T19:42:39Z

WITH is already a reserved keyword in ANSI mode in Spark like the following. I think we don't need to do anything here.

https://github.com/apache/spark/blame/master/docs/sql-keywords.md#L291

peter-toth · 2019-07-10T20:00:51Z

WITH is already a reserved keyword in ANSI mode in Spark like the following. I think we don't need to do anything here.

https://github.com/apache/spark/blame/master/docs/sql-keywords.md#L291

Thanks! Does that mean that we should turn on spark.sql.parser.ansi.enabled=true in with.sql? Or shall I just leave a note about it?

dongjoon-hyun · 2019-07-10T20:21:57Z

Please wrap those create statements with set spark.sql.parser.ansi.enabled=true and set spark.sql.parser.ansi.enabled=false. That will be enough. We don't want to break the existing workload. I think we don't need TODO or other comments since spark.sql.parser.ansi.enabled has a clean meaning.

dongjoon-hyun · 2019-07-10T20:30:09Z

sql/core/src/test/resources/sql-tests/results/pgSQL/with.sql.out

+
+== SQL ==
+create table foo (with baz)
+-----------------------^^^


With set spark.sql.parser.ansi.enabled=true, this will raise the following instead of DataType baz is not supported error.

Error in query: no viable alternative at input 'with'(line 1, pos 18) == SQL == create table foo (with baz) ------------------^^^

SparkQA · 2019-07-10T21:11:54Z

Test build #107472 has finished for PR 24860 at commit 4a356e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/test/resources/sql-tests/inputs/pgSQL/with.sql

dongjoon-hyun · 2019-07-11T02:10:52Z

@maropu . Could you file a JIRA for that PosrgreSQL test should run with the following?

set spark.sql.crossJoin.enabled=true;
set spark.sql.parser.ansi.enabled=true;

maropu · 2019-07-11T06:36:14Z

~~Looks nice, will do now~~
oh, @wangyum seems to be working on it.

SparkQA · 2019-07-11T07:05:02Z

Test build #107515 has finished for PR 24860 at commit 615f592.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2019-07-11T07:06:38Z

retest this please

SparkQA · 2019-07-11T10:34:18Z

Test build #107520 has finished for PR 24860 at commit 615f592.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-07-12T05:54:33Z

Hi, @maropu and @wangyum , and @peter-toth .
Since the followings seems to need more time, I'll merge this first. Please rebase and update those PRs.

[SPARK-28343][SQL][TEST] Enabling cartesian product and ansi mode for PostgreSQL testing #25109 ([SPARK-28343][SQL][TEST] PostgreSQL test should enable cartesian product )
[SPARK-28349][SQL] Add FALSE and SETMINUS to ansiNonReserved #25114 ([SPARK-28349][SQL] Add FALSE and SETMINUS to ansiNonReserved)

dongjoon-hyun

+1, LGTM. Merged to master.

peter-toth · 2019-07-12T06:33:52Z

Thanks @dongjoon-hyun, @maropu for the review!

## What changes were proposed in this pull request? This PR is to port with.sql from PostgreSQL regression tests. https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/with.sql The expected results can be found in the link: https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/expected/with.out When porting the test cases, found 7 PostgreSQL specific features that do not exist in Spark SQL: - [SPARK-19799](https://issues.apache.org/jira/browse/SPARK-19799) Support WITH clause in subqueries - [SPARK-24497](https://issues.apache.org/jira/browse/SPARK-24497) Support recursive SQL query - [SPARK-28297](https://issues.apache.org/jira/browse/SPARK-28297) Handling outer links in CTE subquery expressions - [SPARK-28296](https://issues.apache.org/jira/browse/SPARK-28296) Improved VALUES support - [SPARK-28146](https://issues.apache.org/jira/browse/SPARK-28146) Support IS OF type predicate - [SPARK-28147](https://issues.apache.org/jira/browse/SPARK-28147) Support RETURNING clause - [SPARK-27878](https://issues.apache.org/jira/browse/SPARK-27878) Support ARRAY(sub-SELECT) expressions Also, found one inconsistent behavior: - [SPARK-28299](https://issues.apache.org/jira/browse/SPARK-28299) Evaluation of multiple CTE uses Also, added the following notes: - Spark SQL doesn't support DELETE statement - Spark SQL doesn't support UPDATE statement - Spark SQL doesn't support RULEs - Spark SQL doesn't support UNIQUE constraints - Spark SQL doesn't support ON CONFLICT clause - Spark SQL doesn't support TRIGGERs - Spark SQL doesn't support INHERITS clause ## How was this patch tested? N/A Closes apache#24860 from peter-toth/SPARK-28034. Authored-by: Peter Toth <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun added TEST SQL TESTS and removed TEST labels Jun 13, 2019

dongjoon-hyun mentioned this pull request Jun 15, 2019

[SPARK-28002][SQL] Support WITH clause column aliases #24842

Closed

peter-toth force-pushed the SPARK-28034 branch 2 times, most recently from 40bc3f0 to fd8abe0 Compare June 25, 2019 14:16

peter-toth commented Jun 25, 2019

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/pgSQL/with.sql Outdated Show resolved Hide resolved

peter-toth changed the title ~~[WIP][SPARK-28034][SQL] Port with.sql~~ [SPARK-28034][SQL] Port with.sql Jun 25, 2019

dongjoon-hyun reviewed Jun 26, 2019

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/pgSQL/with.sql Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jul 4, 2019

View reviewed changes

sql/core/src/test/resources/sql-tests/results/pgSQL/with.sql.out Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jul 4, 2019

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/pgSQL/with.sql Outdated Show resolved Hide resolved

peter-toth mentioned this pull request Jul 4, 2019

[SPARK-28251][SQL] Fix error message of inserting into a non-existing table #25054

Closed

peter-toth added 5 commits July 5, 2019 10:15

[SPARK-28034][SQL] Add with.sql

bdabe00

analysis and regenerate output

e2c2fa0

fix review findings

8c12fce

update with.sql source

aa2242c

rebase onto master and regenerate expected results

ad7a86c

peter-toth force-pushed the SPARK-28034 branch from 292be80 to ad7a86c Compare July 5, 2019 08:29

peter-toth added 2 commits July 8, 2019 14:17

recheck and regenerate expected results

5b9997e

minor fix

4a356e5

dongjoon-hyun reviewed Jul 8, 2019

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/pgSQL/with.sql Outdated Show resolved Hide resolved

dongjoon-hyun changed the title ~~[SPARK-28034][SQL] Port with.sql~~ [SPARK-28034][SQL][TEST] Port with.sql Jul 9, 2019

dongjoon-hyun reviewed Jul 10, 2019

View reviewed changes

maropu reviewed Jul 11, 2019

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/pgSQL/with.sql Show resolved Hide resolved

remove todos, set ansi enabled

615f592

dongjoon-hyun approved these changes Jul 12, 2019

View reviewed changes

dongjoon-hyun closed this in fe22faa Jul 12, 2019

[SPARK-28034][SQL][TEST] Port with.sql #24860

[SPARK-28034][SQL][TEST] Port with.sql #24860

Uh oh!

Conversation

peter-toth commented Jun 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

peter-toth commented Jun 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

peter-toth commented Jun 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jun 26, 2019

Uh oh!

SparkQA commented Jun 26, 2019

Uh oh!

Uh oh!

SparkQA commented Jun 28, 2019

Uh oh!

dongjoon-hyun commented Jul 4, 2019

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Jul 4, 2019

Uh oh!

SparkQA commented Jul 4, 2019

Uh oh!

dongjoon-hyun commented Jul 4, 2019

Uh oh!

peter-toth commented Jul 5, 2019

Uh oh!

SparkQA commented Jul 5, 2019

Uh oh!

peter-toth commented Jul 8, 2019

Uh oh!

Uh oh!

dongjoon-hyun commented Jul 8, 2019

Uh oh!

SparkQA commented Jul 8, 2019

Uh oh!

peter-toth commented Jul 8, 2019

Uh oh!

SparkQA commented Jul 8, 2019

Uh oh!

dongjoon-hyun commented Jul 10, 2019

Uh oh!

dongjoon-hyun commented Jul 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peter-toth commented Jul 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jul 10, 2019

Uh oh!

peter-toth commented Jul 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jul 10, 2019

Uh oh!

dongjoon-hyun Jul 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 10, 2019

Uh oh!

Uh oh!

dongjoon-hyun commented Jul 11, 2019

Uh oh!

maropu commented Jul 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Jul 11, 2019

Uh oh!

maropu commented Jul 11, 2019

Uh oh!

SparkQA commented Jul 11, 2019

peter-toth commented Jun 13, 2019 •

edited

Loading

peter-toth commented Jun 13, 2019 •

edited

Loading

peter-toth commented Jun 25, 2019 •

edited

Loading

dongjoon-hyun commented Jul 10, 2019 •

edited

Loading

peter-toth commented Jul 10, 2019 •

edited

Loading

peter-toth commented Jul 10, 2019 •

edited

Loading

dongjoon-hyun Jul 10, 2019 •

edited

Loading

maropu commented Jul 11, 2019 •

edited

Loading