Skip to content

[SPARK-35035][PYTHON] Port Koalas internal implementation unit tests into PySpark #32137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
xinrong-meng wants to merge 5 commits into from
Closed

[SPARK-35035][PYTHON] Port Koalas internal implementation unit tests into PySpark #32137

xinrong-meng wants to merge 5 commits into from

Conversation

@xinrong-meng
Copy link
Member

@xinrong-meng xinrong-meng commented Apr 13, 2021 •
edited
Loading

What changes were proposed in this pull request?

Now that we merged the Koalas main code into the PySpark code base (#32036), we should port the Koalas internal implementation unit tests to PySpark.

Why are the changes needed?

Currently, the pandas-on-Spark modules are not tested fully. We should enable the internal implementation unit tests.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Enable internal implementation unit tests.

Keyword:SPARK-34849

@SparkQA
Copy link

SparkQA commented Apr 13, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41839/

@SparkQA
Copy link

SparkQA commented Apr 13, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41839/

@SparkQA
Copy link

SparkQA commented Apr 13, 2021

Test build #137260 has finished for PR 32137 at commit 0436734.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@xinrong-meng xinrong-meng changed the title [WIP][SPARK-35035][PYTHON] Port Koalas internal implementation unit tests into PySpark [SPARK-35035][PYTHON] Port Koalas internal implementation unit tests into PySpark Apr 13, 2021
@xinrong-meng xinrong-meng marked this pull request as ready for review April 13, 2021 15:35
@xinrong-meng
Copy link
Member Author

xinrong-meng commented Apr 13, 2021

CC @ueshin @HyukjinKwon @itholic

xinrong-meng
xinrong-meng commented Apr 13, 2021
View reviewed changes
dev/sparktestsupport/modules.py
"pyspark.pandas.typedef.typehints",
# unittests
"pyspark.pandas.tests.test_dataframe",
"pyspark.pandas.tests.test_config",
Copy link
Member Author

@xinrong-meng xinrong-meng Apr 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May I sort these tests in my last PR? That would make resolving other PRs' conflicts much easier.

@SparkQA
Copy link

SparkQA commented Apr 13, 2021

Test build #137297 has started for PR 32137 at commit 5860f8a.

@SparkQA
Copy link

SparkQA commented Apr 13, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41877/

@SparkQA
Copy link

SparkQA commented Apr 13, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41877/

ueshin
ueshin approved these changes Apr 13, 2021
View reviewed changes
Copy link
Member

@ueshin ueshin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pending tests.

@SparkQA
Copy link

SparkQA commented Apr 13, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41888/

@SparkQA
Copy link

SparkQA commented Apr 13, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41888/

@SparkQA
Copy link

SparkQA commented Apr 14, 2021

Test build #137308 has finished for PR 32137 at commit cf58ab2.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

itholic
itholic approved these changes Apr 14, 2021
View reviewed changes
Copy link
Contributor

@itholic itholic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Apr 14, 2021

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ueshin ueshin ueshin approved these changes

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

@itholic itholic itholic approved these changes

Assignees

No one assigned

Labels

BUILD CORE PYTHON

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xinrong-meng @SparkQA @HyukjinKwon @ueshin @itholic