BigQuery: jobs created with run_async_query() and run with begin() ignores flag dry_run=True

[sean185]

The aim is to be able to dry run the queries to check how much it would cost us before we actually proceed with it. Hopefully we will be able to use it as a safety mechanism.

I hope I am not misunderstanding the availability of the dry_run parameter as listed on the docs. I have set it, and it appears on the configs, but it runs anyway.

1. Ubuntu 14.04

2. Python 2.7.6

3. google-cloud-python version

google-api-python-client==1.5.5
google-auth==0.5.0
google-auth-httplib2==0.0.2
google-cloud-bigquery==0.22.0
google-cloud-core==0.22.1
google-cloud-storage==0.22.0
googleapis-common-protos==1.5.0

4. Stacktrace if available
   NIL

5. Steps to reproduce

• create job with run_async_query()
• set job.dry_run = True
• run it with job.begin()
• examine job when it is done

6. Code example

job = client.run_async_query(JOB_NAME, QUERY_STRING)
job.dry_run = True
job.begin()
print job.__dict__

[dhermes]

@tseaver We should probably add __slots__ at this point. There have just been too many of these is it accepted / isn't it accepted issues.

[tseaver]

@dhermes This isn't that: we actually do have a dry_run property defined, and we copy it to the job configuration if set.

[dhermes]

Interesting. What do you think is the issue here?

[tseaver]

@dhermes We might have a bug on our side (e.g., copying to the wrong part of the job config resource, or with the wrong name), or the back-end might be ignoring the dryRun flag.

[dhermes]

Easy enough to add a job.dry_run = True before calling .begin() in a system test and adding a handy-dandy import pdb; pdb.set_trace() when api_request() actually sends off a payload?

[sean185]

Hi, yes I am referring to an issue like @tseaver has pointed out, that the dry_run configuration was accepted (was mentioned and fixed in an earlier issue, I had checked before posting). I attempted to read and trace the code to where the configurations were being passed around, but unfortunately I don't think I'm quite familiar enough.

I tried using pdb as suggested, and I got to a point where I saw this:

/home/dev-sean/bq_ops/lib/python2.7/site-packages/google/cloud/bigquery/job.py(1080)_build_resource()->{'configuration': {'query': {'dryRun': True, 'query': 'SELECT * FRO...x.__TABLES__]'}}, 'jobReference': {'jobId': 'debug-dry_ru...-880691b3d08b', 'projectId': u'test-b...ery-1043'}}

According to the documentation here:
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.dryRun

The dryRun property should be parked under configuration and not under query.

Is this perhaps what @dhermes is hoping to find?

[dhermes]

@sean185 Almost certainly what I was hoping for.

[sean185]

@dhermes @tseaver
I don't know if this is relevant, but on the docs for Jobs > QueryJob
https://googlecloudplatform.github.io/google-cloud-python/stable/bigquery-job.html#google.cloud.bigquery.job.QueryJob.dry_run

I found that it mentioned this link for the documentation regarding dryRun,
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.query.dryRun

when the hyperlink should have in fact been
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.dryRun

as per the nesting specified in the docs.

[tseaver]

@dhermes I wonder if we've been gaslighted: note the language describing the option:

[Optional] If set, don't actually run this job. A valid query will return a mostly empty response with some processing statistics, while an invalid query will return the same error it would if it wasn't a dry run. Behavior of non-query jobs is undefined.

Five will get you ten they "promoted" that option up from the query section without telling anybody, or rev'ing the API.

[tseaver]

@sean185 Thanks for digging into that! We need to update to write the flag at the config level, rather than config/query.

[sean185]

@tseaver no problem, just glad that it wasn't a case of me running blind.

So just to clarify, it is this chunk of code, the _build_resource function that places the dry_run variable too deep, correct?

[tseaver]

@sean185 Yes, it calls (eventually) _populate_config_booleans, which does the actual work.

[sean185]

@tseaver ah okay, got it! Will be looking forward to how this issue will be managed, thank you guys for the great work!

[daspecster]

@tseaver @dhermes have one of you guys started on this already? If not I can make a PR.