BUG: Fix using "inf"/"-inf" in na_values for csv with int index column #22169

Templarrr · 2018-08-02T11:56:34Z

closes OverflowError in read_csv when specifying certain na_values #17128
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

The issue happens when you try to use 'inf' or '-inf' as part of na_values in read_csv.

Code snippet to reproduce:

from StringIO import StringIO

import pandas as pd

dataset = StringIO('''index,col1,col2,col3
1,6,10,14
2,7,11,15
3,8,12,16
4,9,13,17
5,inf,-inf,bla
''')
na_values = ['inf', '-inf', 'bla']

df = pd.read_csv(dataset, na_values=na_values, index_col='index')
print df
print df.dtypes

Without fix:

Traceback (most recent call last):
  File "/home/modintsov/workspace/DataRobot/playground.py", line 39, in <module>
    df = pd.read_csv(dataset, na_values=na_values, index_col='index')
  File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 678, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 446, in _read
    data = parser.read(nrows)
  File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 1036, in read
    ret = self._engine.read(nrows)
  File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 1922, in read
    index, names = self._make_index(data, alldata, names)
  File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 1426, in _make_index
    index = self._agg_index(index)
  File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 1520, in _agg_index
    arr, _ = self._infer_types(arr, col_na_values | col_na_fvalues)
  File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/io/parsers.py", line 1600, in _infer_types
    mask = algorithms.isin(values, list(na_values))
  File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/core/algorithms.py", line 418, in isin
    values, _, _ = _ensure_data(values, dtype=dtype)
  File "/home/modintsov/.virtualenvs/dev/local/lib/python2.7/site-packages/pandas/core/algorithms.py", line 82, in _ensure_data
    return _ensure_int64(values), 'int64', 'int64'
  File "pandas/_libs/algos_common_helper.pxi", line 3227, in pandas._libs.algos.ensure_int64
  File "pandas/_libs/algos_common_helper.pxi", line 3232, in pandas._libs.algos.ensure_int64
OverflowError: cannot convert float infinity to integer

With fix (as expected):

       col1  col2  col3
index
1       6.0  10.0  14.0
2       7.0  11.0  15.0
3       8.0  12.0  16.0
4       9.0  13.0  17.0
5       NaN   NaN   NaN
col1    float64
col2    float64
col3    float64
dtype: object

pep8speaks · 2018-08-02T11:56:36Z

Hello @Templarrr! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on August 09, 2018 at 08:44 Hours UTC

codecov · 2018-08-02T16:25:41Z

Codecov Report

Merging #22169 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #22169      +/-   ##
==========================================
- Coverage   92.07%   92.07%   -0.01%     
==========================================
  Files         169      169              
  Lines       50684    50683       -1     
==========================================
- Hits        46668    46666       -2     
- Misses       4016     4017       +1

Flag	Coverage Δ
#multiple	`90.48% <100%> (-0.01%)`	⬇️
#single	`42.34% <100%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/algorithms.py	`94.69% <100%> (ø)`	⬆️
pandas/core/indexes/multi.py	`95.25% <0%> (-0.09%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3bcc2bb...e91a840. Read the comment docs.

jreback · 2018-08-02T17:23:37Z

xref #10065
closes #17128

jreback · 2018-08-02T17:23:54Z

I believe this closes #17128 can you confirm and update the whatsnew to reflect

jreback · 2018-08-02T17:24:44Z

doc/source/whatsnew/v0.23.4.txt

@@ -41,7 +41,7 @@ Bug Fixes

 **Indexing**

-
+- Fix OverflowError when trying to use 'inf' as na_value with int index column


add the issue number here, use double-backticks around OverflowError. use double backticks on na_value; spell out int -> integer.

jreback · 2018-08-02T17:25:02Z

pandas/tests/io/parser/na_values.py

@@ -369,3 +369,13 @@ def test_no_na_filter_on_index(self):
        expected = DataFrame({"a": [1, 4], "c": [3, 6]},
                             index=Index([np.nan, 5.0], name="b"))
        tm.assert_frame_equal(out, expected)
+
+    def test_inf_na_values_with_int_index(self):
+        data = "idx,col1,col2\n1,3,4\n2,inf,-inf"


add the issue number here as a comment

Templarrr · 2018-08-02T18:35:52Z

Oh, wow, it's actually my colleague :)
Didn't know Liau Yung Siang reported that already.

Thanks for the review, it's already late In my timezone, but I'll update the PR tomorrow!

# Conflicts: # doc/source/whatsnew/v0.23.4.txt

Templarrr · 2018-08-03T08:12:00Z

@jreback I've confirmed that this solves the error that @YS-L reported in #17128 and addressed your PR review comments. Can you look again?

Also yesterday the 0.23.4 changelog got updated with today's date, but I don't see on pypi newer pandas version released yet, so I'm not entirely sure - will my bugfix be a part of 0.23.4 or should I move the comment to 0.24.0 changelog?

Templarrr · 2018-08-03T09:42:22Z

The failure in travis is some kind of network glitch, unrelated to these changes :(

Templarrr · 2018-08-06T07:42:40Z

@jreback I see 0.23.4 was released, so this fix obviously didn't make it :) I've moved the changelog line to 0.24.0

Templarrr · 2018-08-07T11:28:03Z

@jreback I see you've moved the label to 0.23.5, didn't know there was 0.23.5 planned :)
There is no 0.23.5 changelog currently, should I create a new one and move my changelog line there?

jreback · 2018-08-07T12:48:11Z

yeah we might do a 0.23.5. if you want to push a 0.23.5 whats (empty) pls do , but a new PR pls.

jreback · 2018-08-08T10:54:56Z

doc/source/whatsnew/v0.24.0.txt

@@ -607,6 +607,7 @@ Indexing
 - Fixed ``DataFrame[np.nan]`` when columns are non-unique (:issue:`21428`)
 - Bug when indexing :class:`DatetimeIndex` with nanosecond resolution dates and timezones (:issue:`11679`)
 - Bug where indexing with a Numpy array containing negative values would mutate the indexer (:issue:`21867`)
+- Fix ``OverflowError`` when trying to use 'inf' as ``na_value`` with integer index column (:issue:`17128`)


can you move to 0.23.5 (rebase on master to see the whatsnew)

will do in few minutes

@jreback done.

jreback

ping on green.

jreback · 2018-08-09T00:08:32Z

doc/source/whatsnew/v0.23.5.txt

@@ -40,3 +40,7 @@ Bug Fixes

 -
 -
+


can you put this in the IO section, and referencde :func:read_csv here

jreback · 2018-08-09T10:38:03Z

thanks @Templarrr

…with int index column

Templarrr · 2018-08-09T11:02:46Z

You're welcome! Always glad to help :)

…with int index column (#22259)

pandas-dev#22169)

Michael Odintsov added 3 commits August 2, 2018 12:19

BUG: Fix using "inf"/"-inf" in na_values for csv with int index column

7331c22

Add the test case for inf as na

29f0992

Changelog

65f83d6

Linter

32c3d75

jreback requested changes Aug 2, 2018

View reviewed changes

jreback added Numeric Operations Arithmetic, Comparison, and Logical operations IO CSV read_csv, to_csv labels Aug 2, 2018

jreback added this to the 0.23.4 milestone Aug 2, 2018

Michael Odintsov added 3 commits August 3, 2018 10:58

Address PR review comments

d39d552

Merge remote-tracking branch 'upstream/master'

bcecbdd

# Conflicts: # doc/source/whatsnew/v0.23.4.txt

Resolve merge conflict

d29d14d

Michael Odintsov added 2 commits August 6, 2018 10:40

Merge remote-tracking branch 'upstream/master'

69cdf5c

Move changelog line to 24.0

bf768f6

jreback modified the milestones: 0.23.4, 0.23.5 Aug 6, 2018

Merge remote-tracking branch 'upstream/master'

d4db49d

jreback requested changes Aug 8, 2018

View reviewed changes

Michael Odintsov added 2 commits August 8, 2018 15:42

Merge remote-tracking branch 'upstream/master'

9b9b550

Move changelog line to 23.5

d1de93a

jreback approved these changes Aug 9, 2018

View reviewed changes

jreback requested changes Aug 9, 2018

View reviewed changes

Michael Odintsov added 2 commits August 9, 2018 11:41

Merge remote-tracking branch 'upstream/master'

0621014

Modify changelog line from PR review feedback

e91a840

jreback approved these changes Aug 9, 2018

View reviewed changes

jreback merged commit 020e948 into pandas-dev:master Aug 9, 2018

lumberbot-app bot pushed a commit that referenced this pull request Aug 9, 2018

Backport PR #22169: BUG: Fix using "inf"/"-inf" in na_values for csv …

13a190e

…with int index column

meeseeksmachine mentioned this pull request Aug 9, 2018

Backport PR #22169 on branch 0.23.x #22259

Merged

jreback pushed a commit that referenced this pull request Aug 9, 2018

Backport PR #22169: BUG: Fix using "inf"/"-inf" in na_values for csv …

faa1992

…with int index column (#22259)

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

BUG: Fix using "inf"/"-inf" in na_values for csv with int index column (

749bd50

pandas-dev#22169)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Fix using "inf"/"-inf" in na_values for csv with int index column #22169

BUG: Fix using "inf"/"-inf" in na_values for csv with int index column #22169

Templarrr commented Aug 2, 2018 •

edited

Loading

pep8speaks commented Aug 2, 2018 •

edited

Loading

codecov bot commented Aug 2, 2018 •

edited

Loading

jreback commented Aug 2, 2018

jreback commented Aug 2, 2018

jreback Aug 2, 2018

jreback Aug 2, 2018

Templarrr commented Aug 2, 2018

Templarrr commented Aug 3, 2018

Templarrr commented Aug 3, 2018

Templarrr commented Aug 6, 2018

Templarrr commented Aug 7, 2018

jreback commented Aug 7, 2018

jreback Aug 8, 2018

Templarrr Aug 8, 2018

Templarrr Aug 8, 2018

jreback left a comment

jreback Aug 9, 2018

jreback commented Aug 9, 2018

Templarrr commented Aug 9, 2018

BUG: Fix using "inf"/"-inf" in na_values for csv with int index column #22169

BUG: Fix using "inf"/"-inf" in na_values for csv with int index column #22169

Conversation

Templarrr commented Aug 2, 2018 • edited Loading

pep8speaks commented Aug 2, 2018 • edited Loading

Comment last updated on August 09, 2018 at 08:44 Hours UTC

codecov bot commented Aug 2, 2018 • edited Loading

Codecov Report

jreback commented Aug 2, 2018

jreback commented Aug 2, 2018

jreback Aug 2, 2018

Choose a reason for hiding this comment

jreback Aug 2, 2018

Choose a reason for hiding this comment

Templarrr commented Aug 2, 2018

Templarrr commented Aug 3, 2018

Templarrr commented Aug 3, 2018

Templarrr commented Aug 6, 2018

Templarrr commented Aug 7, 2018

jreback commented Aug 7, 2018

jreback Aug 8, 2018

Choose a reason for hiding this comment

Templarrr Aug 8, 2018

Choose a reason for hiding this comment

Templarrr Aug 8, 2018

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback Aug 9, 2018

Choose a reason for hiding this comment

jreback commented Aug 9, 2018

Templarrr commented Aug 9, 2018

Templarrr commented Aug 2, 2018 •

edited

Loading

pep8speaks commented Aug 2, 2018 •

edited

Loading

codecov bot commented Aug 2, 2018 •

edited

Loading