[SPARK-34886][PYTHON] Port/integrate Koalas DataFrame unit test into PySpark #32083

xinrong-meng · 2021-04-08T02:16:58Z

What changes were proposed in this pull request?

Now that we merged the Koalas main code into the PySpark code base (#32036), we should port the Koalas DataFrame unit test to PySpark.

Why are the changes needed?

Currently, the pandas-on-Spark modules are not tested at all. We should enable the DataFrame unit test first.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Enable the DataFrame unit test.

Keyword:SPARK-34849

ueshin · 2021-04-08T02:34:04Z

ok to test.

SparkQA · 2021-04-08T04:34:45Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41627/

SparkQA · 2021-04-08T06:47:03Z

Test build #137049 has finished for PR 32083 at commit 3b924c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-04-08T11:57:54Z

add to whitelist

HyukjinKwon · 2021-04-08T11:58:35Z

python/pyspark/pandas/testing/utils.py

+from pyspark.pandas.utils import default_session, sql_conf as sqlc, SPARK_CONF_ARROW_ENABLED
+
+
+class SQLTestUtils(object):


I think we have this util in PySpark. we should probably merge

That's a good idea!

May I take it as a separate task later and port test files first?

Shall we file a JIRA ticket to track the task then?

Certainly, I filed https://issues.apache.org/jira/browse/SPARK-34999 to track this.

SparkQA · 2021-04-08T13:10:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41654/

SparkQA · 2021-04-08T13:10:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41654/

SparkQA · 2021-04-08T14:45:18Z

Test build #137076 has finished for PR 32083 at commit 3b924c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-08T17:18:47Z

Test build #137092 has finished for PR 32083 at commit 6d13af9.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-08T18:18:11Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41670/

SparkQA · 2021-04-08T18:18:12Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41670/

SparkQA · 2021-04-08T20:14:40Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41675/

ueshin · 2021-04-08T21:21:44Z

I guess we should add pyspark.pandas.tests to heavy_tests in python/run-tests.py?

ueshin

LGTM, pending tests.

SparkQA · 2021-04-08T22:01:29Z

Test build #137097 has finished for PR 32083 at commit 4abe527.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-08T22:30:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41678/

SparkQA · 2021-04-08T22:30:51Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41678/

SparkQA · 2021-04-09T00:55:07Z

Test build #137100 has finished for PR 32083 at commit 3a73eba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-09T01:34:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41685/

SparkQA · 2021-04-09T01:34:46Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41685/

SparkQA · 2021-04-09T03:20:47Z

Test build #137107 has finished for PR 32083 at commit f5d9fd3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-04-09T03:54:36Z

retest this please

SparkQA · 2021-04-09T05:06:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41692/

SparkQA · 2021-04-09T05:10:28Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41692/

SparkQA · 2021-04-09T06:14:43Z

Test build #137113 has finished for PR 32083 at commit f5d9fd3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-04-09T06:30:23Z

retest this please

HyukjinKwon · 2021-04-09T06:48:04Z

I manually checked that all related tests passed during running multiple times. I don't believe this cause any extra test failures.

Merged to master.

SparkQA · 2021-04-09T08:10:41Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41698/

SparkQA · 2021-04-09T08:20:24Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41698/

SparkQA · 2021-04-09T09:35:46Z

Test build #137119 has finished for PR 32083 at commit f5d9fd3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

xinrong-meng added 2 commits April 7, 2021 17:47

test_dataframe; testing folder

4a9eac0

Add test_dataframe to pyspark_pandas module

3b924c0

xinrong-meng changed the title ~~[WIP] [SPARK-34886] Port/integrate Koalas DataFrame unit test into PySpark~~ [WIP][SPARK-34886][PYTHON] Port/integrate Koalas DataFrame unit test into PySpark Apr 8, 2021

github-actions bot added BUILD CORE PYTHON labels Apr 8, 2021

HyukjinKwon reviewed Apr 8, 2021

View reviewed changes

Skip test_to_markdown if tabulate not installed

6d13af9

Add noqa

4abe527

pyspark.pandas.tests as heavy_tests

3a73eba

ueshin approved these changes Apr 8, 2021

View reviewed changes

xinrong-meng changed the title ~~[WIP][SPARK-34886][PYTHON] Port/integrate Koalas DataFrame unit test into PySpark~~ [SPARK-34886][PYTHON] Port/integrate Koalas DataFrame unit test into PySpark Apr 8, 2021

xinrong-meng marked this pull request as ready for review April 8, 2021 22:03

HyukjinKwon approved these changes Apr 8, 2021

View reviewed changes

- extra import

f5d9fd3

HyukjinKwon closed this in 3af2c1b Apr 9, 2021

		from pyspark.pandas.utils import default_session, sql_conf as sqlc, SPARK_CONF_ARROW_ENABLED


		class SQLTestUtils(object):

[SPARK-34886][PYTHON] Port/integrate Koalas DataFrame unit test into PySpark #32083

[SPARK-34886][PYTHON] Port/integrate Koalas DataFrame unit test into PySpark #32083

Uh oh!

Conversation

xinrong-meng commented Apr 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

ueshin commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

HyukjinKwon commented Apr 8, 2021

Uh oh!

HyukjinKwon Apr 8, 2021

Choose a reason for hiding this comment

Uh oh!

xinrong-meng Apr 8, 2021

Choose a reason for hiding this comment

Uh oh!

ueshin Apr 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xinrong-meng Apr 9, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

ueshin commented Apr 8, 2021

Uh oh!

ueshin left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 8, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

HyukjinKwon commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

HyukjinKwon commented Apr 9, 2021

Uh oh!

HyukjinKwon commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

Uh oh!

SparkQA commented Apr 9, 2021

xinrong-meng commented Apr 8, 2021 •

edited

Loading

ueshin Apr 8, 2021 •

edited

Loading