Skip to content

Conversation

@max-sixty
Copy link
Contributor

Now we have clustering, we're looking at running more shorter queries. This PR reduces the volume of logs in INFO, particularly for short queries.

Existing output:

2018-08-24 18:32:43,618 - pandas_gbq.gbq - INFO - Requesting query... 
2018-08-24 18:32:44,068 - pandas_gbq.gbq - INFO - ok.
Query running...
2018-08-24 18:32:44,073 - pandas_gbq.gbq - INFO - Job ID: d2c20d3f-97d3-44f8-b1be-ee8b10ec2818
Query running...
2018-08-24 18:32:44,700 - pandas_gbq.gbq - INFO - Got 1 rows.

2018-08-24 18:32:44,725 - pandas_gbq.gbq - INFO - Total time taken 1.11 s.
Finished at 2018-08-24 18:32:44.

New output for queries < 7s:

Query running...
Query running...

(it's repeated because we send an additional query, as per #198)

New output for queries > 7s:

Query running...
Query running...
2018-08-24 18:32:44,725 - pandas_gbq.gbq - INFO - Total time taken 11.87 s.
Finished at 2018-08-24 18:32:44.

Most of the existing output can be restored by setting the debug level to DEBUG.

Ideally I think we'd replace all this with tqdm, but this is a quick starter

@max-sixty max-sixty requested a review from tswast August 24, 2018 18:54
Copy link
Collaborator

@tswast tswast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the plan to use tqdm for most of these use cases.

@max-sixty max-sixty merged commit 9e623fb into googleapis:master Aug 25, 2018
@max-sixty max-sixty deleted the logging branch August 25, 2018 17:37
@max-sixty
Copy link
Contributor Author

@tswast what do you think about removing the Query running... log for queries under 5s?

We're starting to roll out short queries internally and even with these changes, that's making up > 50% of our logs. We could disable the logger internally, but others may think similarly.

@tswast
Copy link
Collaborator

tswast commented Sep 12, 2018

@tswast what do you think about removing the Query running... log for queries under 5s?

I like that idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants