"403 Exceeded rate limits: too many table update operations for this table" when uploading just 2 hours every hour.

[ghost]

• Programming language: Python
• OS: Linux / Google Cloud Function
• Language runtime version: 3.7
• Package version: 0.14.1

Error message:
pandas_gbq.gbq.GenericGBQException: Reason: 403 Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas

I am uploading a very small amount of rows (less than 20) to a BQ table every hour through a cloud function running on a schedule. I'm receiving the error quite frequently. There are no other scripts writing to that table.
I get the error after a log that says:

2 out of 2 rows loaded.
1 out of 1 rows loaded.

Both those logs are for a single upload command.

[max-sixty]

Is this specific to pandas-gbq? That sounds like a general BQ error

[ghost]

I'm only using pandas-gbq to to push rows to a bq table.

[tswast]

Can you share more about your workload? Is the schema changing?

How many to_gbq calls is your script making? Is it one per row or are all the rows bundled together into a single dataframe?

[tswast]

You'll have the best results if you can minimize the number of calls to to_gbq. It uses BigQuery "load jobs" which have a limited quota, though once an hour should be well within that.

[ghost]

I don't recall which of my pipeline this was. I believe I moved on to Google's Datastore because of the Rate limit errors.
What I can tell you is that the schema was not changing. I was adding to a BigQuery table every hour or so, about 20 rows that I was getting from an API.
Sadly there is no way for me to reproduce this ATM.

[tswast]

Thanks for getting back to us.

There is a feature request open at #300 to use the streaming API for writes, which avoids some rate limits with load jobs (but at the expense of some complexity around writing to new/recreated tables). This hasn't been implemented in pandas-gbq yet, but is available in the google-cloud-bigquery library with the insert_rows_from_dataframe method.

I'll close this issue as it isn't reproducible, but feel free to follow #300 or open a new issue if you have a reproducible example.