`to_gbq` requires `bigquery.tables.create` permission even with if_exists="append"

[acarmel]

I have a workload that runs with a service account that has permissions only to write to an existing table but not to create one or change the schema. I encountered a problem that even though the table exists and we passed the argument if_exists="append" we got a 403 error that the service account doesn't have bigquery.tables.create permission, though it shouldn't be needed at all.

After investigating it I found the problem. In the LoadJobConfig sent with the request there is a field createDisposition that if it's not set the default is CREATE_IF_NEEDED and that requires the service account which submits the jobs to have the bigquery.tables.create permission.

I suggest setting the field createDisposition to CREATE_NEVER to prevent the need to have this permission.

There is no fear that this will restrict users who want to_gbq to create the table for them if it doesn't exist because at the state the LoadJobConfig is created the table must already exists.

Steps to reproduce

1. Create a service account with these permissions

   • roles/bigquery.dataViewer
   • custom role with:
     bigquery.datasets.get
     bigquery.tables.get
     bigquery.tables.updateData

2. Create a table

3. Run this code

dataframe = pandas.DataFrame({"test": ["some test data"]})

pandas_gbq.to_gbq(
    dataframe=dataframe,
    destination_table="my_dataset.my_table_name",
    project_id="my_project_id",
    if_exists="append",
)

Stack trace

    pandas_gbq.to_gbq(
  File "/lib/python3.10/site-packages/pandas_gbq/gbq.py", line 1198, in to_gbq
    connector.load_data(
  File "/lib/python3.10/site-packages/pandas_gbq/gbq.py", line 610, in load_data
    self.process_http_error(ex)
  File "/lib/python3.10/site-packages/pandas_gbq/gbq.py", line 386, in process_http_error
    raise GenericGBQException("Reason: {0}".format(ex))
pandas_gbq.exceptions.GenericGBQException: Reason: 403 POST https://bigquery.googleapis.com/upload/bigquery/v2/projects/my_project_id/jobs?uploadType=multipart: Access Denied: Dataset my_project_id:my_dataset: Permission bigquery.tables.create denied on dataset my_project_id:my_dataset (or it may not exist).

[tswast]

#539 looks like a reasonable fix to me. Thanks for bringing this to our attention!