read_csv(compression='gzip') fails while reading compressed file with tf.gfile.GFile in Python 2

[wmitsuda]

Code Sample, a copy-pastable example if possible

Sample (1)

import tensorflow as tf
import pandas as pd
with tf.gfile.GFile('test.csv') as f: pd.read_csv(f)

Sample (2)

import tensorflow as tf
import pandas as pd
with tf.gfile.GFile('test.csv.gz') as f: pd.read_csv(f, compression='gzip')

Problem description

I'm converting some code to run on Google Cloud, and in the process I'm changing the way my datasets are read. I started using tf.gfile.GFile implementation from Tensorflow, as it is portable and can read both local files and files from storage buckets.

Also in the process I'm changing my code to work with Python 2 instead of Python 3.

Not sure if it is a bug in Pandas or Tensorflow code, but this issue seems similar to #14222, so I'm opening an issue here first.

To reproduce, create two files: test.csv and test.csv.gz locally. Run both samples in python 2. The sample (1) works fine, but sample (2) crashes with an error: "AttributeError: 'NoneType' object has no attribute 'Tell'"

Strangely, both samples work fine in python 3. I'm using the same library versions in python 2 and 3: pandas 0.19.2 and tensorflow 1.0.

Expected Output

Both samples should work in python 2.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 2.7.13.final.0 python-bits: 64 OS: Darwin OS-release: 16.5.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: None.None

pandas: 0.19.2
nose: None
pip: 9.0.1
setuptools: 32.1.0
Cython: None
numpy: 1.12.0
scipy: 0.19.0
statsmodels: None
xarray: None
IPython: 5.3.0
sphinx: None
patsy: None
dateutil: 2.5.2
pytz: 2016.10
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 2.0.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: 0.10.3
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
boto: None
pandas_datareader: None

[TomAugspurger]

Can you try with master? I think this may have been fixed by #16150

[wmitsuda]

Tested with master, it is still failing with the same error message.

pandas: 0.21.0.dev+8.gd50f981d7

[TomAugspurger]

At a glance, it looks like gfile.GFile doesn't follow python's IO interface for seek:

In [28]: type(g)
Out[28]: tensorflow.python.platform.gfile.GFile

In [29]: g.seek?
Signature: g.seek(position)
Docstring: Seeks to the position in the file.
File:      ~/Envs/py27/lib/python2.7/site-packages/tensorflow/python/lib/io/file_io.py
Type:      instancemethod

.seek is supposed to take an optional whence offset. https://docs.python.org/3/library/io.html#io.IOBase.seek
Can you take a look at what's going on in tensorflow?

[wmitsuda]

@TomAugspurger could you please comment on the related issue on Tensorflow? They are asking questions about pandas which I have no means to comment on.

[TomAugspurger]

Sure, before I do though can you run this code and post your trackeback?

import pandas as pd
import numpy as np
import tensorflow as tf

df = pd.DataFrame(np.random.randn(10, 2))
df.to_csv("foo.csv.gz", compression="gzip")

pd.read_csv("foo.csv.gz", compression="gzip")

with tf.gfile.GFile("foo.csv.gz") as gf:
    pd.read_csv(gf, compression="gzip")

I get

Traceback (most recent call last):
  File "bug.py", line 11, in <module>
    pd.read_csv(gf, compression="gzip")
  File "/Users/taugspurger/Envs/py27/lib/python2.7/site-packages/pandas/pandas/io/parsers.py", line 655, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/Users/taugspurger/Envs/py27/lib/python2.7/site-packages/pandas/pandas/io/parsers.py", line 405, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Users/taugspurger/Envs/py27/lib/python2.7/site-packages/pandas/pandas/io/parsers.py", line 762, in __init__
    self._make_engine(self.engine)
  File "/Users/taugspurger/Envs/py27/lib/python2.7/site-packages/pandas/pandas/io/parsers.py", line 966, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/Users/taugspurger/Envs/py27/lib/python2.7/site-packages/pandas/pandas/io/parsers.py", line 1582, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 562, in pandas._libs.parsers.TextReader.__cinit__ (pandas/_libs/parsers.c:6175)
  File "pandas/_libs/parsers.pyx", line 751, in pandas._libs.parsers.TextReader._get_header (pandas/_libs/parsers.c:9268)
  File "pandas/_libs/parsers.pyx", line 953, in pandas._libs.parsers.TextReader._tokenize_rows (pandas/_libs/parsers.c:11755)
  File "pandas/_libs/parsers.pyx", line 2173, in pandas._libs.parsers.raise_parser_error (pandas/_libs/parsers.c:28589)
TypeError: seek() takes exactly 2 arguments (3 given)

which is different than the original issue (i'm a bit ahead of 0.20.1 on my python 2 environment).

[wmitsuda]

With pandas 0.19.2 + Tensorflow 1.0.0, I got this:

Traceback (most recent call last):
  File "test.py", line 11, in <module>
    pd.read_csv(gf, compression="gzip")
  File "/usr/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 646, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 389, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/usr/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 730, in __init__
    self._make_engine(self.engine)
  File "/usr/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 923, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/usr/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 1390, in __init__
    self._reader = _parser.TextReader(src, **kwds)
  File "pandas/parser.pyx", line 535, in pandas.parser.TextReader.__cinit__ (pandas/parser.c:6086)
  File "pandas/parser.pyx", line 710, in pandas.parser.TextReader._get_header (pandas/parser.c:8843)
  File "pandas/parser.pyx", line 911, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:11308)
  File "pandas/parser.pyx", line 2014, in pandas.parser.raise_parser_error (pandas/parser.c:26862)
AttributeError: 'NoneType' object has no attribute 'Tell'

With pandas 0.19.2 + Tensorflow 1.1.0 (which was just released), I got a similar trace:

Traceback (most recent call last):
  File "test.py", line 11, in <module>
    pd.read_csv(gf, compression="gzip")
  File "/usr/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 646, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 389, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/usr/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 730, in __init__
    self._make_engine(self.engine)
  File "/usr/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 923, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/usr/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 1390, in __init__
    self._reader = _parser.TextReader(src, **kwds)
  File "pandas/parser.pyx", line 535, in pandas.parser.TextReader.__cinit__ (pandas/parser.c:6086)
  File "pandas/parser.pyx", line 710, in pandas.parser.TextReader._get_header (pandas/parser.c:8843)
  File "pandas/parser.pyx", line 911, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:11308)
  File "pandas/parser.pyx", line 2014, in pandas.parser.raise_parser_error (pandas/parser.c:26862)
TypeError: seek() takes exactly 2 arguments (3 given)

[TomAugspurger]

Ok, that's the difference, I was using tensorflow 1.1.0

[wmitsuda]

Closing as it was a tensorflow bug: tensorflow/tensorflow#9806