Storage: Bucket.list_blobs(max_results=n) does not behave as documented

[bz2]

The max_results parameter of list_blobs() is documented as controlling the maximum number of blobs returned in each page of results, but actually limits the total number of results as the name implies.

Compare the Bucket.list_blobs() documentation:
https://googleapis.dev/python/storage/latest/buckets.html#google.cloud.storage.bucket.Bucket.list_blobs

max_results (int) – The maximum number of blobs in each page of results from this request. Non-positive values are ignored. Defaults to a sensible value set by the API.

With the Iterator documentation:
https://googleapis.dev/python/google-api-core/latest/page_iterator.html#google.api_core.page_iterator.Iterator

max_results (int) – The maximum number of results to fetch.

Also the implementation of HTTPIterator which is used by list_blobs() internally does treat max_results as a hard limit for total num_results:
https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/page_iterator.py#L378

Code example

iterator = some_big_bucket.list_blobs(max_results=100)
assert len(list(iterator)) > 100  # throws
assert sum(len(list(page)) for page in iterator.pages) > 100  # throws

Suggested resolution

Change the documentation to match what the parameter actually does. If supplying a paging size is required, a new argument to HTTPIterator could be added and exposed up through the list_blobs() interface.

[frankyn]

Hi folks,

I dug into the Page iterator and found that max_results is strict in the total number of results to retrieve. However, I think there's a usability issue here in the case you want to limit the number of items per page while retrieving all items.

Given the expectation is set in the code, moving forward with the PR that exists makes sense for now because it would require adding support for this path.

The workaround for this is the following:

from google.cloud import storage

storage_client = storage.Client()
bucket = storage_client.bucket("bucket-name")

blobs = bucket.list_blobs(max_results=3)
start = False
while blobs.next_page_token or start is False:
   for blob in blobs:
       print(blob.name)
   blobs = bucket.list_blobs(max_results=3, page_token=blobs.next_page_token)
   start = True

@crwilcox this is a usability issue because it doesn't fully expose the GCS API behavior when setting maxResults.