data-science-onramp.data-cleaning.clean_test: test_clean failed

[flaky-bot]

Note: #5312 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

---

commit: 16941a1
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/data-science-onramp/data-cleaning/clean_test.py", line 172, in test_clean
    result = operation.result()
  File "/workspace/data-science-onramp/data-cleaning/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/future/polling.py", line 134, in result
    raise self._exception
google.api_core.exceptions.FailedPrecondition: 400 Google Cloud Dataproc Agent reports job failure. If logs are available, they can be found at 'https://console.cloud.google.com/dataproc/jobs/04df2dd7-79ab-4441-b123-3fab37009703?project=python-docs-samples-tests®ion=us-central1' and in 'gs://dataproc-28f7d075-c5ac-4938-a5ad-65eafd8032d3-us-central1/google-cloud-dataproc-metainfo/79ff181d-2abf-48d4-8190-6444e9354cbe/jobs/04df2dd7-79ab-4441-b123-3fab37009703/driveroutput'.

[leahecole]

@bradmiro is helping me understand this rn

[leahecole]

testing locally to see if I can replicate and if it has to do with the pyspark connector jar we specify when submitting the dataproc job

[leahecole]

Was able to replicate - am now trying running 3.7 as is, and 3.6 with gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.18.1.jar on submission

[leahecole]

Known issue being tracked in b/181278351