Skip to content

REG: Fix read_parquet from file-like objects #34500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jun 12, 2020
9 changes: 4 additions & 5 deletions pandas/io/parquet.py
Original file line number Diff line number Diff line change
Expand Up @@ -122,11 +122,10 @@ def write(
file_obj_or_path.close()

def read(self, path, columns=None, **kwargs):
parquet_ds = self.api.parquet.ParquetDataset(
path, filesystem=get_fs_for_path(path), **kwargs
)
kwargs["columns"] = columns
result = parquet_ds.read_pandas(**kwargs).to_pandas()
kwargs["use_pandas_metadata"] = True
result = self.api.parquet.read_table(
path, columns=columns, filesystem=get_fs_for_path(path), **kwargs
).to_pandas()
return result


Expand Down
8 changes: 8 additions & 0 deletions pandas/tests/io/test_parquet.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
""" test parquet compat """
import datetime
from distutils.version import LooseVersion
from io import BytesIO
import os
from warnings import catch_warnings

Expand Down Expand Up @@ -567,6 +568,13 @@ def test_s3_roundtrip_for_dir(self, df_compat, s3_resource, pa, partition_col):
repeat=1,
)

def test_file_like_obj_support(self, df_compat):
buffer = BytesIO()
df_compat.to_parquet(buffer)
df_from_buf = pd.read_parquet(buffer)
print(df_from_buf)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't checkin print statements?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep thanks - I’m about to tidy this up.

tm.assert_frame_equal(df_compat, df_from_buf)

def test_partition_cols_supported(self, pa, df_full):
# GH #23283
partition_cols = ["bool", "int"]
Expand Down