[SPARK-34052][SQL] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT" #31107

sunchao · 2021-01-10T00:41:59Z

What changes were proposed in this pull request?

This passes original SQL text to CacheTableAsSelect command in DSv1 and v2 so that it will be stored instead of the analyzed logical plan, similar to CREATE VIEW command.

In addition, this changes the behavior of dropping temporary view to also invalidate dependent caches in a cascade, when the config SQLConf.STORE_ANALYZED_PLAN_FOR_VIEW is false (which is the default value).

Why are the changes needed?

Currently, after creating a temporary view with CACHE TABLE ... AS SELECT command, the view can still be queried even after the source table is dropped or replaced (in v2). This can cause correctness issue.

For instance, in the following:

> CREATE TABLE t ...;
> CACHE TABLE v AS SELECT * FROM t;
> DROP TABLE t;
> SELECT * FROM v;

The last select query still returns the old (and stale) result instead of fail. Note that the cache is already invalidated as part of dropping table t, but the temporary view v still exist.

On the other hand, the following:

> CREATE TABLE t ...;
> CREATE TEMPORARY VIEW v AS SELECT * FROM t;
> CACHE TABLE v;
> DROP TABLE t;
> SELECT * FROM v;

will throw "Table or view not found" error in the last select query.

This is related to #30567 which aligns the behavior of temporary view and global view by storing the original SQL text for temporary view, as opposed to the analyzed logical plan. However, the PR only handles CreateView case but not the CacheTableAsSelect case.

This also changes uncache logic and use cascade invalidation for temporary views created above. This is to align its behavior to how a permanent view is handled as of today, and also to avoid potential issues where a dependent view becomes invalid while its data is still kept in cache.

Does this PR introduce any user-facing change?

Yes, now when SQLConf.STORE_ANALYZED_PLAN_FOR_VIEW is set to false (the default value), whenever a table/permanent view/temp view that a cached view depends on is dropped, the cached view itself will become invalid during analysis, i.e., user will get "Table or view not found" error. In addition, when the dependent is a temp view in the previous case, the cache itself will also be invalidated.

How was this patch tested?

Modified/Enhanced some existing tests.

SparkQA · 2021-01-10T02:18:30Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38469/

SparkQA · 2021-01-10T02:51:29Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38469/

SparkQA · 2021-01-10T03:09:11Z

Test build #133880 has finished for PR 31107 at commit 69c81c6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-01-11T17:00:43Z

Could you take a look at the UT failure, @sunchao ?

sunchao · 2021-01-11T18:03:22Z

@dongjoon-hyun yes I'm looking at some of the test failures. Will update this PR soon.

sunchao · 2021-01-11T22:07:28Z

One of the test failure:

CachedTableSuite.CACHE TABLE tableName AS SELECT ...
org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to qualifier on unresolved object, tree: 'key

looks like a separate bug related to #30567, and I've opened SPARK-34076 to track it.

SparkQA · 2021-01-14T02:35:20Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38619/

SparkQA · 2021-01-14T03:07:17Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38619/

SparkQA · 2021-01-14T06:49:23Z

Test build #134032 has finished for PR 31107 at commit bce8265.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2021-01-14T19:57:52Z

Also, cc @MaxGekk

SparkQA · 2021-01-14T20:38:58Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38651/

SparkQA · 2021-01-14T21:11:46Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38651/

SparkQA · 2021-01-15T00:23:49Z

Test build #134065 has finished for PR 31107 at commit 380bbb1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sunchao · 2021-01-15T03:54:17Z

cc @cloud-fan @viirya @HyukjinKwon

sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CacheTableExec.scala

sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala

cloud-fan · 2021-01-15T04:56:18Z

thanks for catching this bug!

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala

SparkQA · 2021-01-15T09:43:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38680/

SparkQA · 2021-01-15T10:09:12Z

Test build #134094 has finished for PR 31107 at commit 120dd3d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-01-15T10:16:44Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38680/

SparkQA · 2021-01-15T18:42:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38702/

viirya · 2021-01-19T18:10:32Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala

+          catalog.getTempViewOrPermanentTableMetadata(tableName).viewText.isDefined
        sparkSession.sharedState.cacheManager.uncacheQuery(
-          sparkSession.table(tableName), cascade = !isTempView)
+          sparkSession.table(tableName), cascade = !isTempView || hasViewText)


Is the PR title and description up-to-dated? Looks like the change in the PR is basically changing the uncaching behavior of view with view text.

Looks like we already did cascade uncache when dropping a source table here, so I'm wondering why the title is "A cached view should become invalid after the source table is dropped"?

Yes the PR title is not very good - I just updated it. I think the description is more aligned to what the PR does but I'll try to add more details there.

viirya · 2021-01-19T19:03:53Z

sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala

+          // dropping a temp view trigger cache invalidation on dependents iff the config is
+          // turned on


I think the comment is incorrect, right? It should be when the config is turned off.

Nice catch! I confused myself by considering "turned on" = "turned on storing SQL text"

viirya · 2021-01-19T19:07:28Z

CREATE TABLE t ...;
CACHE TABLE v AS SELECT * FROM t;
DROP TABLE t;
SELECT * FROM v;

BTW, for the example in the description, I think the t should be a view to show the issue?

sunchao · 2021-01-19T19:25:33Z

I think it doesn't matter whether t is a table or view. The issue is v is stored as analyzed plan which doesn't get re-analyzed when t is dropped.

viirya · 2021-01-19T19:30:31Z

Hmm, but seems I don't see such example is added as a test?

sunchao · 2021-01-19T19:33:01Z

I used to have a test here but removed it since I think it largely overlaps with some existing ones, but yeah I could add it back if you think that helps.

viirya · 2021-01-19T19:38:58Z

yeah, thanks, i think it is better to have it as it is used in the description as example.

SparkQA · 2021-01-19T19:53:24Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38820/

SparkQA · 2021-01-19T20:41:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38823/

SparkQA · 2021-01-19T20:42:57Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38820/

viirya

lgtm, thanks.

SparkQA · 2021-01-19T21:09:50Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38823/

SparkQA · 2021-01-19T23:37:12Z

Test build #134235 has finished for PR 31107 at commit 12cd598.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-01-20T00:42:47Z

Test build #134237 has finished for PR 31107 at commit 40ef4b1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-01-20T01:06:47Z

Didn't take a close look too but sounds making sense.

cloud-fan · 2021-01-20T02:09:37Z

thanks, merging to master!

cloud-fan · 2021-01-20T02:11:43Z

@sunchao can you open a backport PR for 3.1 since the new SQL temp view semantic is already there? Thanks!

sunchao · 2021-01-20T04:52:46Z

Thanks for the review!

@sunchao can you open a backport PR for 3.1 since the new SQL temp view semantic is already there? Thanks!

I got lots of conflicts when backporting it - @imback82 is there any plan to backport SPARK-33654 to branch-3.1? I see its affected version is 3.1.0.

cloud-fan · 2021-01-20T05:04:49Z

Ah 3.1 doesn't have CacheTableAsSelect. We need to re-implement this fix with the old CacheTableCommand.

sunchao · 2021-01-20T05:08:26Z

I see. Let me implement it separately for branch 3.1.

…E TABLE .. AS SELECT" This passes original SQL text to `CacheTableAsSelect` command in DSv1 and v2 so that it will be stored instead of the analyzed logical plan, similar to `CREATE VIEW` command. In addition, this changes the behavior of dropping temporary view to also invalidate dependent caches in a cascade, when the config `SQLConf.STORE_ANALYZED_PLAN_FOR_VIEW` is false (which is the default value). Currently, after creating a temporary view with `CACHE TABLE ... AS SELECT` command, the view can still be queried even after the source table is dropped or replaced (in v2). This can cause correctness issue. For instance, in the following: ```sql > CREATE TABLE t ...; > CACHE TABLE v AS SELECT * FROM t; > DROP TABLE t; > SELECT * FROM v; ``` The last select query still returns the old (and stale) result instead of fail. Note that the cache is already invalidated as part of dropping table `t`, but the temporary view `v` still exist. On the other hand, the following: ```sql > CREATE TABLE t ...; > CREATE TEMPORARY VIEW v AS SELECT * FROM t; > CACHE TABLE v; > DROP TABLE t; > SELECT * FROM v; ``` will throw "Table or view not found" error in the last select query. This is related to apache#30567 which aligns the behavior of temporary view and global view by storing the original SQL text for temporary view, as opposed to the analyzed logical plan. However, the PR only handles `CreateView` case but not the `CacheTableAsSelect` case. This also changes uncache logic and use cascade invalidation for temporary views created above. This is to align its behavior to how a permanent view is handled as of today, and also to avoid potential issues where a dependent view becomes invalid while its data is still kept in cache. Yes, now when `SQLConf.STORE_ANALYZED_PLAN_FOR_VIEW` is set to false (the default value), whenever a table/permanent view/temp view that a cached view depends on is dropped, the cached view itself will become invalid during analysis, i.e., user will get "Table or view not found" error. In addition, when the dependent is a temp view in the previous case, the cache itself will also be invalidated. Modified/Enhanced some existing tests. Closes apache#31107 from sunchao/SPARK-34052. Lead-authored-by: Chao Sun <[email protected]> Co-authored-by: Chao Sun <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…"CACHE TABLE .. AS SELECT ..." This is a backport of #31107 to branch-3.1. ### What changes were proposed in this pull request? This passes original SQL text to `CacheTableCommand` command in DSv1 so that it will be stored instead of the analyzed logical plan, similar to `CREATE VIEW` command. In addition, this changes the behavior of dropping temporary view to also invalidate dependent caches in a cascade, when the config `SQLConf.STORE_ANALYZED_PLAN_FOR_VIEW` is false (which is the default value). ### Why are the changes needed? Currently, after creating a temporary view with `CACHE TABLE ... AS SELECT` command, the view can still be queried even after the source table is dropped or replaced (in v2). This can cause correctness issue. For instance, in the following: ```sql > CREATE TABLE t ...; > CACHE TABLE v AS SELECT * FROM t; > DROP TABLE t; > SELECT * FROM v; ``` The last select query still returns the old (and stale) result instead of fail. Note that the cache is already invalidated as part of dropping table `t`, but the temporary view `v` still exist. On the other hand, the following: ```sql > CREATE TABLE t ...; > CREATE TEMPORARY VIEW v AS SELECT * FROM t; > CACHE TABLE v; > DROP TABLE t; > SELECT * FROM v; ``` will throw "Table or view not found" error in the last select query. This is related to #30567 which aligns the behavior of temporary view and global view by storing the original SQL text for temporary view, as opposed to the analyzed logical plan. However, the PR only handles `CreateView` case but not the `CacheTableAsSelect` case. This also changes uncache logic and use cascade invalidation for temporary views created above. This is to align its behavior to how a permanent view is handled as of today, and also to avoid potential issues where a dependent view becomes invalid while its data is still kept in cache. ### Does this PR introduce _any_ user-facing change? Yes, now when `SQLConf.STORE_ANALYZED_PLAN_FOR_VIEW` is set to false (the default value), whenever a table/permanent view/temp view that a cached view depends on is dropped, the cached view itself will become invalid during analysis, i.e., user will get "Table or view not found" error. In addition, when the dependent is a temp view in the previous case, the cache itself will also be invalidated. ### How was this patch tested? Added new test cases. Also modified and enhanced some existing related tests. Closes #31300 from sunchao/SPARK-34052-branch-3.1. Authored-by: Chao Sun <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

…E TABLE .. AS SELECT" ### What changes were proposed in this pull request? This passes original SQL text to `CacheTableAsSelect` command in DSv1 and v2 so that it will be stored instead of the analyzed logical plan, similar to `CREATE VIEW` command. In addition, this changes the behavior of dropping temporary view to also invalidate dependent caches in a cascade, when the config `SQLConf.STORE_ANALYZED_PLAN_FOR_VIEW` is false (which is the default value). ### Why are the changes needed? Currently, after creating a temporary view with `CACHE TABLE ... AS SELECT` command, the view can still be queried even after the source table is dropped or replaced (in v2). This can cause correctness issue. For instance, in the following: ```sql > CREATE TABLE t ...; > CACHE TABLE v AS SELECT * FROM t; > DROP TABLE t; > SELECT * FROM v; ``` The last select query still returns the old (and stale) result instead of fail. Note that the cache is already invalidated as part of dropping table `t`, but the temporary view `v` still exist. On the other hand, the following: ```sql > CREATE TABLE t ...; > CREATE TEMPORARY VIEW v AS SELECT * FROM t; > CACHE TABLE v; > DROP TABLE t; > SELECT * FROM v; ``` will throw "Table or view not found" error in the last select query. This is related to apache#30567 which aligns the behavior of temporary view and global view by storing the original SQL text for temporary view, as opposed to the analyzed logical plan. However, the PR only handles `CreateView` case but not the `CacheTableAsSelect` case. This also changes uncache logic and use cascade invalidation for temporary views created above. This is to align its behavior to how a permanent view is handled as of today, and also to avoid potential issues where a dependent view becomes invalid while its data is still kept in cache. ### Does this PR introduce _any_ user-facing change? Yes, now when `SQLConf.STORE_ANALYZED_PLAN_FOR_VIEW` is set to false (the default value), whenever a table/permanent view/temp view that a cached view depends on is dropped, the cached view itself will become invalid during analysis, i.e., user will get "Table or view not found" error. In addition, when the dependent is a temp view in the previous case, the cache itself will also be invalidated. ### How was this patch tested? Modified/Enhanced some existing tests. Closes apache#31107 from sunchao/SPARK-34052. Lead-authored-by: Chao Sun <[email protected]> Co-authored-by: Chao Sun <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…ade for temp views ### What changes were proposed in this pull request? This PR includes the following changes: 1. in `CatalogImpl.uncacheTable`, invalidate caches in cascade when the target table is a temp view, and `spark.sql.legacy.storeAnalyzedPlanForView` is false (default value). 2. make `SessionCatalog.lookupTempView` public and return processed temp view plan (i.e., with `View` op). ### Why are the changes needed? Following [SPARK-34052](https://issues.apache.org/jira/browse/SPARK-34052) (#31107), we should invalidate in cascade for `CatalogImpl.uncacheTable` when the table is a temp view, so that the behavior is consistent. ### Does this PR introduce _any_ user-facing change? Yes, now `SQLContext.uncacheTable` will drop temp view in cascade by default. ### How was this patch tested? Added a UT Closes #31462 from sunchao/SPARK-34347. Authored-by: Chao Sun <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]>

github-actions bot added the SQL label Jan 10, 2021

sunchao changed the title ~~[SPARK-34052][SQL] A cached view should become invalid after a table is dropped~~ [SPARK-34052][SQL] A cached view should become invalid after the source table is dropped Jan 10, 2021

sunchao mentioned this pull request Jan 10, 2021

[SPARK-34039][SQL] ReplaceTable should invalidate cache #31081

Closed

sunchao marked this pull request as draft January 12, 2021 18:16

sunchao mentioned this pull request Jan 13, 2021

[SPARK-34076][SQL] SQLContext.dropTempTable fails if cache is non-empty #31136

Closed

sunchao force-pushed the SPARK-34052 branch from 69c81c6 to bce8265 Compare January 14, 2021 01:40

sunchao marked this pull request as ready for review January 14, 2021 22:38

cloud-fan reviewed Jan 15, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Jan 15, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CacheTableExec.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Jan 15, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala Outdated Show resolved Hide resolved

viirya reviewed Jan 15, 2021

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala Outdated Show resolved Hide resolved

viirya reviewed Jan 19, 2021

View reviewed changes

Add comments

12cd598

sunchao changed the title ~~[SPARK-34052][SQL] A cached view should become invalid after the source table is dropped~~ [SPARK-34052][SQL] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT" Jan 19, 2021

viirya reviewed Jan 19, 2021

View reviewed changes

address more comments

40ef4b1

viirya approved these changes Jan 19, 2021

View reviewed changes

cloud-fan closed this in 902a08b Jan 20, 2021

sunchao mentioned this pull request Jan 22, 2021

[SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..." #31300

Closed

sunchao mentioned this pull request Feb 3, 2021

[SPARK-34347][SQL] CatalogImpl.uncacheTable should invalidate in cascade for temp views #31462

Closed

		// dropping a temp view trigger cache invalidation on dependents iff the config is
		// turned on

[SPARK-34052][SQL] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT" #31107

[SPARK-34052][SQL] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT" #31107

Uh oh!

Conversation

sunchao commented Jan 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Jan 10, 2021

Uh oh!

SparkQA commented Jan 10, 2021

Uh oh!

SparkQA commented Jan 10, 2021

Uh oh!

dongjoon-hyun commented Jan 11, 2021

Uh oh!

sunchao commented Jan 11, 2021

Uh oh!

sunchao commented Jan 11, 2021

Uh oh!

SparkQA commented Jan 14, 2021

Uh oh!

SparkQA commented Jan 14, 2021

Uh oh!

SparkQA commented Jan 14, 2021

Uh oh!

dongjoon-hyun commented Jan 14, 2021

Uh oh!

SparkQA commented Jan 14, 2021

Uh oh!

SparkQA commented Jan 14, 2021

Uh oh!

SparkQA commented Jan 15, 2021

Uh oh!

sunchao commented Jan 15, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cloud-fan commented Jan 15, 2021

Uh oh!

Uh oh!

SparkQA commented Jan 15, 2021

Uh oh!

SparkQA commented Jan 15, 2021

Uh oh!

SparkQA commented Jan 15, 2021

Uh oh!

SparkQA commented Jan 15, 2021

Uh oh!

viirya Jan 19, 2021

Choose a reason for hiding this comment

Uh oh!

sunchao Jan 19, 2021

Choose a reason for hiding this comment

Uh oh!

viirya Jan 19, 2021

Choose a reason for hiding this comment

Uh oh!

sunchao Jan 19, 2021

Choose a reason for hiding this comment

Uh oh!

viirya commented Jan 19, 2021

Uh oh!

sunchao commented Jan 19, 2021

Uh oh!

viirya commented Jan 19, 2021

Uh oh!

sunchao commented Jan 19, 2021

Uh oh!

viirya commented Jan 19, 2021

Uh oh!

SparkQA commented Jan 19, 2021

Uh oh!

SparkQA commented Jan 19, 2021

Uh oh!

SparkQA commented Jan 19, 2021

sunchao commented Jan 10, 2021 •

edited

Loading