Investigate novel storage system test failures

[dhermes]

1. Bad error response (non-JSON)

ERROR: test_copy_existing_file (storage.TestStorageWriteFiles)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 181, in test_copy_existing_file
    new_blob = self.bucket.copy_blob(blob, self.bucket, 'CloudLogoCopy')
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/storage/bucket.py", line 460, in copy_blob
    method='POST', path=api_path, _target_object=new_blob)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/connection.py", line 337, in api_request
    error_info=method + ' ' + url)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/exceptions.py", line 185, in make_exception
    payload = json.loads(content)
  File "/opt/python/2.7.9/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/opt/python/2.7.9/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/python/2.7.9/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

From https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/84536058
2. Missing files

======================================================================
ERROR: tearDownModule (storage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 47, in tearDownModule
    bucket.delete(force=True)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/bucket.py", line 358, in delete
    _target_object=None)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/connection.py", line 419, in api_request
    error_info=method + ' ' + url)
gcloud.exceptions.Conflict: 409 The bucket you tried to delete was not empty. (DELETE https://www.googleapis.com/storage/v1/b/new1440625507341)
======================================================================
FAIL: test_second_level (storage.TestStoragePseudoHierarchy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 290, in test_second_level
    'parent/child/file22.txt'])
AssertionError: Lists differ: [] != ['parent/child/file21.txt', 'parent/child/file22.txt']
Second list contains 2 additional elements.
First extra element 0:
parent/child/file21.txt
- []

From https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/77405118
3. Backend Error (500)

======================================================================
ERROR: test_list_buckets (storage.TestStorageBuckets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 58, in tearDown
    CLIENT.bucket(bucket_name).delete()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/batch.py", line 263, in __exit__
    self.finish()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/batch.py", line 249, in finish
    self._finish_futures(responses)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/batch.py", line 231, in _finish_futures
    raise make_exception(*exception_args)
gcloud.exceptions.InternalServerError: 500 Backend Error
----------------------------------------------------------------------

From:

1. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/77600321
2. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/82969179
3. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/97488225
4. Delete blob -> 404 (probably eventual consistency error)

======================================================================
ERROR: setUpClass (storage.TestStoragePseudoHierarchy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/regression/storage.py", line 241, in setUpClass
    blob.delete()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/regression3/lib/python3.4/site-packages/gcloud/storage/blob.py", line 264, in delete
    return self.bucket.delete_blob(self.name)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/regression3/lib/python3.4/site-packages/gcloud/storage/bucket.py", line 361, in delete_blob
    self.connection.api_request(method='DELETE', path=blob_path)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/regression3/lib/python3.4/site-packages/gcloud/connection.py", line 293, in api_request
    raise make_exception(response, content)
gcloud.exceptions.NotFound: 404 Not Found
----------------------------------------------------------------------

From:

1. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/58811668
2. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/97491709
3. Eventual consistency error

======================================================================
FAIL: test_paginate_files (storage.TestStorageListFiles)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 231, in test_paginate_files
    self.assertEqual(len(last_blobs), truncation_size)
AssertionError: 0 != 1

From:

1. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/84165044
2. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/92143108
3. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/93053612
4. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/97374670
5. https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/98356544
6. Service unavailable (503) error

ERROR: test_list_buckets (storage.TestStorageBuckets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/storage.py", line 76, in test_list_buckets
    bucket = CLIENT.create_bucket(bucket_name)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/client.py", line 215, in create_bucket
    bucket.create(client=self)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/storage/bucket.py", line 172, in create
    data={'name': self.name}, _target_object=self)
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests3/lib/python3.4/site-packages/gcloud/connection.py", line 337, in api_request
    error_info=method + ' ' + url)
gcloud.exceptions.ServiceUnavailable: 503 Backend Error (POST https://www.googleapis.com/storage/v1/b?project=precise-truck-742)

From: https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/90171219

[tseaver]

7. Conflict deleting bucket in module teardown:

======================================================================
ERROR: tearDownModule (storage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/system_tests/storage.py", line 55, in tearDownModule
    Config.TEST_BUCKET.delete(force=True)
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/storage/bucket.py", line 358, in delete
    _target_object=None)
  File "/home/tseaver/projects/agendaless/Google/src/gcloud-python/.tox/system-tests/lib/python2.7/site-packages/gcloud/connection.py", line 347, in api_request
    error_info=method + ' ' + url)
Conflict: 409 The bucket you tried to delete was not empty. (DELETE https://www.googleapis.com/storage/v1/b/new_1470678826358)

----------------------------------------------------------------------

Found during a local run of tox -e system-tests.

[dhermes]

That's from the eventual consistency of listing objects. Using force=True

• gets a list of all objects / blobs
• deletes each object / blob
• deletes the bucket

but if the list isn't truly comprehensive, the bucket deletion will fail with the 409 conflict you showed. Luckily just trying bucket.delete(force=True) again will eventually succeed.

[tseaver]

4. and 7) addressed via 00d42d8.

[dhermes]

@tseaver Since these reports are stale, let's just close out and then address as new failures occur