Skip to content

[SPARK-35366][SQL] Avoid using deprecated buildForBatch and buildForStreaming #32497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
linhongliu-db wants to merge 7 commits into from
Closed

[SPARK-35366][SQL] Avoid using deprecated buildForBatch and buildForStreaming #32497

linhongliu-db wants to merge 7 commits into from

Conversation

@linhongliu-db
Copy link
Contributor

@linhongliu-db linhongliu-db commented May 11, 2021

What changes were proposed in this pull request?

Currently, in DSv2, we are still using the deprecated buildForBatch and buildForStreaming.
This PR implements the build, toBatch, toStreaming interfaces to replace the deprecated ones.

Why are the changes needed?

Code refactor

Does this PR introduce any user-facing change?

No

How was this patch tested?

exsting UT

@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42882/

@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42882/

@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42879/

@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42879/

@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42889/

@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42889/

@SparkQA
Copy link

SparkQA commented May 11, 2021

Test build #138357 has finished for PR 32497 at commit d078953.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42895/

@SparkQA
Copy link

SparkQA commented May 11, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42895/

@SparkQA
Copy link

SparkQA commented May 11, 2021

Test build #138366 has finished for PR 32497 at commit aa51f84.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 11, 2021

Test build #138372 has finished for PR 32497 at commit 41a46f5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@linhongliu-db
Copy link
Contributor Author

linhongliu-db commented May 11, 2021

cc @cloud-fan

cloud-fan
cloud-fan reviewed May 11, 2021
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWrite.scala
formatName: String,
supportsDataType: DataType => Boolean,
info: LogicalWriteInfo) extends WriteBuilder {
trait FileWrite extends Write {
Copy link
Contributor

@cloud-fan cloud-fan May 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

HeartSaVioR
HeartSaVioR reviewed May 12, 2021
View reviewed changes
Copy link
Contributor

@HeartSaVioR HeartSaVioR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK to me. Would be nice to address existing review comments as well.

@linhongliu-db
Copy link
Contributor Author

linhongliu-db commented May 13, 2021

@cloud-fan @HeartSaVioR, thanks for reviewing. comments are addressed.

@SparkQA
Copy link

SparkQA commented May 13, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43029/

@SparkQA
Copy link

SparkQA commented May 13, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43029/

@SparkQA
Copy link

SparkQA commented May 13, 2021

Test build #138509 has finished for PR 32497 at commit e48aee5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

cloud-fan commented May 13, 2021

thanks, merging to master!

@cloud-fan cloud-fan closed this in 6aa2594 May 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HeartSaVioR HeartSaVioR HeartSaVioR left review comments

@cloud-fan cloud-fan cloud-fan approved these changes

Assignees

No one assigned

Labels

AVRO SQL STRUCTURED STREAMING

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@linhongliu-db @SparkQA @cloud-fan @HeartSaVioR