[SPARK-34999][PYTHON] Consolidate PySpark testing utils #32177

xinrong-meng · 2021-04-14T20:37:23Z

What changes were proposed in this pull request?

Consolidate PySpark testing utils by removing python/pyspark/pandas/testing, and then creating a file pandasutils under python/pyspark/testing for test utilities used in pyspark/pandas.

Why are the changes needed?

python/pyspark/pandas/testing hold test utilites for pandas-on-spark, and python/pyspark/testing contain test utilities for pyspark. Consolidating them makes code cleaner and easier to maintain.

Updated import statements are as shown below:

from pyspark.testing.sqlutils import SQLTestUtils
from pyspark.testing.pandasutils import PandasOnSparkTestCase, TestUtils
(PandasOnSparkTestCase is the original ReusedSQLTestCase in python/pyspark/pandas/testing/utils.py)

Minor improvements include:

Usage of missing library's requirement_message
except ImportError rather than except
import pyspark.pandas alias as ps rather than pp

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit tests under python/pyspark/pandas/tests.

Keyword:SPARK-34849

xinrong-meng · 2021-04-14T20:38:08Z

[DONE] This PR ought to be adjusted and merged after #32139 and #32152.

SparkQA · 2021-04-14T20:57:15Z

Test build #137372 has finished for PR 32177 at commit b31ab33.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

github-actions · 2021-04-14T21:24:58Z

Test build #749688774 for PR 32177 at commit b31ab33.

SparkQA · 2021-04-14T21:28:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41950/

SparkQA · 2021-04-14T21:33:39Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41950/

python/pyspark/pandas/testing/utils.py

python/pyspark/testing/utils.py

github-actions · 2021-04-15T02:59:09Z

Test build #750536294 for PR 32177 at commit 9781e5d.

github-actions · 2021-04-15T03:00:03Z

Test build #750538121 for PR 32177 at commit de6cb1e.

SparkQA · 2021-04-15T03:13:17Z

Test build #137392 has finished for PR 32177 at commit 9781e5d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-15T03:55:14Z

Test build #137395 has finished for PR 32177 at commit de6cb1e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-15T03:59:08Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41970/

Yikun

There are some suggestion on inline comments

python/pyspark/pandas/tests/test_dataframe_conversion.py

python/pyspark/testing/utils.py

github-actions · 2021-04-15T16:49:01Z

Test build #752784527 for PR 32177 at commit 6638d40.

github-actions · 2021-04-15T17:27:33Z

Test build #752884941 for PR 32177 at commit c79a606.

github-actions · 2021-04-15T17:29:46Z

Test build #752891640 for PR 32177 at commit 47af208.

SparkQA · 2021-04-15T17:32:21Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42006/

github-actions · 2021-04-15T17:43:55Z

Test build #752927701 for PR 32177 at commit 73ce19a.

github-actions · 2021-04-15T17:47:07Z

Test build #752931779 for PR 32177 at commit aa3590e.

github-actions · 2021-04-15T18:05:50Z

Test build #752986243 for PR 32177 at commit b70bba9.

SparkQA · 2021-04-15T18:16:39Z

Test build #137430 has finished for PR 32177 at commit 73ce19a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-15T18:17:17Z

Test build #137432 has finished for PR 32177 at commit aa3590e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-15T18:30:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42007/

SparkQA · 2021-04-15T18:30:43Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42007/

SparkQA · 2021-04-15T18:44:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42009/

SparkQA · 2021-04-15T18:45:26Z

Test build #137429 has finished for PR 32177 at commit 6638d40.

This patch fails PySpark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2021-04-15T18:49:32Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42009/

SparkQA · 2021-04-22T17:24:16Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42346/

SparkQA · 2021-04-22T17:24:17Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42346/

SparkQA · 2021-04-22T17:41:50Z

Test build #137816 has finished for PR 32177 at commit 95bdbcf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin

LGTM, pending tests.

ueshin · 2021-04-22T20:06:24Z

Thanks! merging to master.

xinrong-meng changed the title ~~[SPARK-34999][PYTHON] Consolidate PySpark testing utils~~ [WIP][SPARK-34999][PYTHON] Consolidate PySpark testing utils Apr 14, 2021

github-actions bot added CORE PYTHON labels Apr 14, 2021

ueshin reviewed Apr 15, 2021

View reviewed changes

Yikun requested changes Apr 15, 2021

View reviewed changes

xinrong-meng force-pushed the port.merge_utils branch from c79a606 to 47af208 Compare April 15, 2021 17:29

xinrong-meng added 22 commits April 22, 2021 09:26

Del folder

11748e6

Format

39f13be

SQLTestUtils in sqlutils only

3eb5f90

Create pandasutils

6c07de3

Use pandasutils

8540ffc

Check libs installed

156b790

except ImportError

b0b4c51

Format

2eb6c87

Restore requirement_message

9500e6f

Use requirement_message

8ceac6e

Clearer note

837d256

Change import alias

5039c64

Adjust for master

2183e0e

Fix import

ab577b4

Fmt

f4be164

Rename ReusedSQLTestCase to PandasOnSparkTestCase

e278bba

Fix

c0547ca

Fix imports

4163452

Fix

4d9ddb0

Fix

a5401ae

Fix

66e66d5

Reorder imports

95bdbcf

xinrong-meng force-pushed the port.merge_utils branch from 8660ae8 to 95bdbcf Compare April 22, 2021 16:27

ueshin approved these changes Apr 22, 2021

View reviewed changes

ueshin closed this in 4d2b559 Apr 22, 2021

[SPARK-34999][PYTHON] Consolidate PySpark testing utils #32177

[SPARK-34999][PYTHON] Consolidate PySpark testing utils #32177

Uh oh!

Conversation

xinrong-meng commented Apr 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

xinrong-meng commented Apr 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Apr 14, 2021

Uh oh!

github-actions bot commented Apr 14, 2021

Uh oh!

SparkQA commented Apr 14, 2021

Uh oh!

SparkQA commented Apr 14, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 15, 2021

Uh oh!

github-actions bot commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

Yikun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 15, 2021

Uh oh!

github-actions bot commented Apr 15, 2021

Uh oh!

github-actions bot commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

github-actions bot commented Apr 15, 2021

Uh oh!

github-actions bot commented Apr 15, 2021

Uh oh!

github-actions bot commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 15, 2021

Uh oh!

SparkQA commented Apr 22, 2021

Uh oh!

SparkQA commented Apr 22, 2021

Uh oh!

SparkQA commented Apr 22, 2021

Uh oh!

ueshin left a comment

Choose a reason for hiding this comment

xinrong-meng commented Apr 14, 2021 •

edited

Loading

xinrong-meng commented Apr 14, 2021 •

edited

Loading