[Python][C++] Enable CSV reader to read from concatenated gzip stream

[asfimport]

If two gzipped files are concatenated together, the result is a valid gzip file.  However, it appears that pyarrow.csv.read_csv will only read the portion related to the first file.

If the repro script here is run, the output is:

$ python repro.py
pyarrow.csv only reads one row:
x
0 1
pandas reads two rows:
x
0 1
1 2
pyarrow version: 0.14.0

Reporter: Jordan Samuels
Assignee: Antoine Pitrou / @pitrou

PRs and other links:

• GitHub Pull Request #4923

_{Note: This issue was originally created as ARROW-5974. Please see the migration documentation for further details.}

[asfimport]

Jordan Samuels:
Confirmed same issue for 0.13.0

[asfimport]

Antoine Pitrou / @pitrou:
I'm not sure Pandas does it deliberately. @wesm Do you know about that?

[asfimport]

Antoine Pitrou / @pitrou:
From a technical point of view, when you're concatenating several gzip files, you're creating a file with several gzip streams. Python reads all the streams as if they were a single one. Arrow doesn't.

Is there a reason why you'd like to this to work?

[asfimport]

Jordan Samuels:
@pitrou there are two reasons I'd like it to work.  Generally, because it seems intuitive that reading a gzip'd file with decompression should have the same end state as gunzip'ing the file on the command line and then reading it - and this is not the case in my example.  Also specifically, I'm working with some third party data that is evidently of this form - a vendor has concatenated gzip'd files together as .gz files, and it would be useful to be able to read them directly with read_csv directly rather than gunzip them as a separate step.

[asfimport]

Wes McKinney / @wesm:
You would need to develop a C++ interface to interpret a file-of-concatenated-gzips as a single logical stream.

I don't think pandas has any specific logic about this, it's all handled by the gzip.GzipFile

https://github.com/pandas-dev/pandas/blob/2811464a87e6e18f5daef87ee700075ebd8a5e7d/pandas/_libs/parsers.pyx#L625

[asfimport]

Wes McKinney / @wesm:
I updated the JIRA to reclassify this as a new feature. PRs welcome

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
Issue resolved by pull request 4923
#4923