[BigQuery] Feature Request - An easy way to get a QueryResults from a QueryJob

[tswast]

Problem:

There isn't an easy way to fetch the row results of a client.run_async_query(job_id, query) call.

I have to do some yucky stuff to get it to work in a sample I'm working on.

client = bigquery.Client()
query_job = client.run_async_query(str(uuid.uuid4()), query)
query_job.use_legacy_sql = False
query_job.begin()

wait_for_job(query_job)

query_results = bigquery.query.QueryResults('', client)
query_results._properties['jobReference'] = {
    'jobId': query_job.name,
    'projectId': query_job.project
}

Possible Solutions:

• Modify constructor for QueryResults or add a class function to take an optional job object.
• Add a method to QueryJob get_query_results() that does this same yuckiness (modifying the internal _properties)

I'm sure there are other solutions, too, which is why I'm asking for input rather than going off and trying to fix right away.

Reference:

• QueryJob: https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/bigquery/job.py#L933
• QueryResults: https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/bigquery/query.py#L42
• Docs: https://raw.githubusercontent.com/GoogleCloudPlatform/gcloud-python/master/docs/bigquery-usage.rst Notice the async query does not show fetching row results.

[tswast]

Note: my current "solution" of modifying the internal _properties works, but it's not something I'd want to include in our docs.

[dhermes]

Also seems a wait_for_job might be something for us to provide?

[tswast]

Nah, not necessarily. wait_for_job is more for the sample. It makes an asynchronous call into a synchronous one. ;-)

def wait_for_job(job):
    while True:
        job.reload()  # Refreshes the state via a GET request.
        if job.state == 'DONE':
            if job.error_result:
                raise RuntimeError(job.error_result)
            return
        time.sleep(1)

This probably isn't the best way to do it, and if someone was using a real asynchronous framework like gevent, twisted, or tornado, they'd probably want to do something slightly different.

[tseaver]

@tswast Thanks for the report! As it sits today, QueryResults is intended to model a synchronous query: it surfaces the job property only once that has been populated from the server during its run() or fetch_data() methods.

It looks as thought we need to split out the part which initiates the synchronous query and the bits which allow fetching the results, to make it clearer how the two are supposed to interact, and how to reuse the second bit with an async query job.

The synchronous case would look something like:

query = client.sync_query(QUERY_STRING)
query.timeout_ms = 30000
query.run()

if not query.complete:
    wait_for_job(query.job)

# if we know the result set is small
for row in query.rows:
    do_something_with(row)

#otherwise, page through them
results = query.results()

rows, total_count, token = results.fetch_data()

while True:
    for row in rows:
        do_something_with(row)
    if token is None:
        break
    rows, total_count, token = results.fetch_data(
        page_token=token)

The asyncronous case would look like:

job = client.async_query(QUERY_STRING)
job.begin()

wait_for_job(job)

results = job.results()

rows, total_count, token = results.fetch_data()

while True:
    for row in rows:
        do_something_with(row)
    if token is None:
        break
    rows, total_count, token = results.fetch_data(
        page_token=token)

[tswast]

job.results() for asynchronous queries sounds good to me.

I don't think we'd need to change how synchronous queries work.

[theacodes]

Bump, any update on this?