GCS failures with latest version

[aarontp]

Hopefully just needs a minor refactor. Note that PSQ also uses Datastore (transparently) to return task results.

It's broken with:
gapic-google-cloud-datastore-v1 (0.15.3)
google-cloud-datastore (1.4.0)
proto-google-cloud-datastore-v1 (0.90.4)

But works with:
gapic-google-cloud-datastore-v1 (0.15.3)
google-cloud-datastore (1.0.0)
grpc-google-cloud-datastore-v1 (0.14.0)
proto-google-cloud-datastore-v1 (0.90.4)

[INFO] Task failed with exception: [request() got an unexpected keyword argument 'data']
[[INFO] Writing /var/tmp/1511642221-817a08dc1d334e4587df1739da220996-PlasoTask/worker-log.txt to GCS path output/1511642221-817a08dc1d334e4587df1739da220996-PlasoTask/worker-log.txt
DEBUG: traceback.format_exc(): Traceback (most recent call last):
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 312, in run_wrapper
result = self.run(evidence, result)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/plaso.py", line 49, in run
ret, result = self.execute(cmd, result, save_files=[plaso_log], close=True)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 257, in execute
result.close(success=True)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 114, in close
self.save_local_file(logfile)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 177, in save_local_file
new_path = writer.write(file_)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/output_writers.py", line 176, in write
blob.upload_from_filename(file_, client=self.client)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 992, in upload_from_filename
size=total_bytes)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 946, in upload_from_file
client, file_obj, content_type, size, num_retries)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 867, in _do_upload
client, stream, content_type, size, num_retries)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 700, in _do_multipart_upload
transport, data, object_metadata, content_type)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/resumable_media/requests/upload.py", line 97, in transmit
retry_strategy=self._retry_strategy)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/resumable_media/requests/_helpers.py", line 101, in http_request
func, RequestsMixin._get_status_code, retry_strategy)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/resumable_media/_helpers.py", line 146, in wait_and_retry
response = func()
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google_auth_httplib2.py", line 198, in request
uri, method, body=body, headers=request_headers, **kwargs)
TypeError: request() got an unexpected keyword argument 'data'

[INFO] Task failed with exception: [request() got an unexpected keyword argument 'data']

[aarontp]

Probably related to:
googleapis/google-cloud-python#3736

[aarontp]

FWIW, same error with 1.2.0

[aarontp]

Turns out this is likely only GCS (I guess I had confused some of the datastore package names), and the packages that are (were?) having dependency problems are:
google-cloud
google-cloud-core
google-cloud-storage
requests

Hopefully we can resolve these issues without breaking our pinning of the google-cloud-pubsub package to 0.26.

[aarontp]

Stub script to reproduce this:

"""
#!/usr/bin/env python

from google.cloud import storage

file_ = '/tmp/test.txt'
open(file_, 'w').write('testing')

c = storage.Client(project='MYPROJECT')
b = c.get_bucket('MYBUCKET')
blob = storage.Blob('output/test.txt', b)
blob.upload_from_filename(file_)
"""

[aarontp]

So, it appears that pinning the pubsub version is what is causing problems (indirectly). When removing the pubsub pinning, the above code runs just fine. Below is the pip freeze diff between two virtualenvs setup with the only difference being the pubsub version being pinned to 0.26. I think the google-cloud-core package is the issue, and is mentioned in the bug referenced above.

$ diff ~/pip.head.freeze.txt ~/pip.pubsub-0.26.freeze.txt
24a26
> google-auth-httplib2==0.0.3
28c30
< google-cloud-core==0.28.0
---
> google-cloud-core==0.25.0
36c38
< google-cloud-pubsub==0.29.2
---
> google-cloud-pubsub==0.26.0

[aarontp]

Ugh, now after pinning google-cloud-core>=0.28.0, something is wrong with psq. Looks like something in the pubsub library? It seems to be the correct version though:

"""
$ pip list | grep google-cloud
gapic-google-cloud-datastore-v1 (0.15.3)
gapic-google-cloud-error-reporting-v1beta1 (0.15.3)
gapic-google-cloud-logging-v2 (0.91.3)
gapic-google-cloud-pubsub-v1 (0.15.4)
google-cloud (0.31.0)
google-cloud-bigquery (0.28.0)
google-cloud-bigtable (0.28.1)
google-cloud-core (0.28.0)
google-cloud-datastore (1.4.0)
google-cloud-dns (0.28.0)
google-cloud-error-reporting (0.28.0)
google-cloud-firestore (0.28.0)
google-cloud-language (1.0.0)
google-cloud-logging (1.4.0)
google-cloud-monitoring (0.28.0)
google-cloud-pubsub (0.26.0)
google-cloud-resource-manager (0.28.0)
google-cloud-runtimeconfig (0.28.0)
google-cloud-spanner (0.29.0)
google-cloud-speech (0.30.0)
google-cloud-storage (1.6.0)
google-cloud-trace (0.16.0)
google-cloud-translate (1.3.0)
google-cloud-videointelligence (1.0.0)
google-cloud-vision (0.28.0)
proto-google-cloud-datastore-v1 (0.90.4)
proto-google-cloud-error-reporting-v1beta1 (0.15.3)
proto-google-cloud-logging-v2 (0.91.3)
proto-google-cloud-pubsub-v1 (0.15.4)

$ pip list | grep requests
requests (2.18.4)

$ python -c 'import psq'
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/buildtools/current/sitecustomize/sitecustomize.py", line 152, in SetupPathsAndImport
return real_import(name, globals, locals, fromlist, level)
File "/lib/python2.7/site-packages/psq/init.py", line 24, in
from .broadcast_queue import BroadcastQueue
File "lib/python2.7/site-packages/psq/broadcast_queue.py", line 18, in
from google.cloud import pubsub
File "lib/python2.7/site-packages/google/cloud/pubsub/init.py", line 30, in
from google.cloud.pubsub.client import Client
File "lib/python2.7/site-packages/google/cloud/pubsub/client.py", line 21, in
from google.cloud.pubsub._http import Connection
File "lib/python2.7/site-packages/google/cloud/pubsub/_http.py", line 25, in
from google.cloud.iterator import HTTPIterator
ImportError: No module named iterator
"""

[aarontp]

Ok, it seems like this file got moved in this PR:
googleapis/google-cloud-python#3770

And then the version is updated to 0.27.0, so I guess I need to try to pin 0.26.0. It breaks with 0.25.0 though, so I don't have high hopes.