BUG: NotFoundException from to_gbq when replacing table in project different from project_id parameter

[wnojopra]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd

df = pd.DataFrame(
  {
    "A": pd.Series(1, index=list(range(2)))
  }
)

# 0) Start with an empty test_dataset. test_table does not exist at this point.


# 1) This successfully creates a table named test_table in dataset test_dataset in project willyn_data_project
df.to_gbq('willyn_data_project.test_dataset.test_table', project_id='willyn_job_project')

# 2) This fails with pandas_gbq.gbq.NotFoundException: Table does not exist
df.to_gbq('willyn_data_project.test_dataset.test_table', project_id='willyn_job_project', if_exists='replace')

Problem description

I am working on a project where researchers can write to a dataset in one GCP project, but must run query jobs in another. In the above code example, willyn_data_project is the name of the project that contains the dataset, and willyn_job_project is the name of the project where query jobs are executed.

When attempting to use if_exists='replace', we get the error pandas_gbq.gbq.NotFoundException: Table does not exist. I believe what is happening is we are attempting to delete a table from willyn_job_project, when we should be deleting a table in willyn_data_project.

Expected Output

No error.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 5f648bf
python : 3.9.2.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.40-1rodete2-amd64
Version : #1 SMP Debian 5.10.40-1rodete2 (2021-06-22)
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.2
numpy : 1.19.5
pytz : 2021.1
dateutil : 2.8.1
pip : 20.3.4
setuptools : 52.0.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : 0.15.0
pyarrow : 3.0.0
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : 0.8.7
xarray : None
xlrd : None
xlwt : None
numba : None

[jreback]

pls file on the pandas-gbq repo as pandas just calls this

[wnojopra]

Thanks. Filed googleapis/python-bigquery-pandas#377