BUG: Raise ValueError if a column index in usecols is out of bounds. … #25686

ghost · 2019-03-12T10:36:18Z

closes ENH/BUG: usecols does not raise an exception when col index is out of bounds. #25623
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

codecov · 2019-03-12T11:14:30Z

Codecov Report

Merging #25686 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #25686      +/-   ##
==========================================
+ Coverage   91.29%   91.29%   +<.01%     
==========================================
  Files         173      173              
  Lines       52961    52970       +9     
==========================================
+ Hits        48350    48359       +9     
  Misses       4611     4611

Flag	Coverage Δ
#multiple	`89.87% <100%> (ø)`	⬆️
#single	`41.73% <20%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/io/parsers.py	`95.43% <100%> (+0.08%)`	⬆️
pandas/util/testing.py	`88.98% <0%> (-0.1%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9185af2...4f92607. Read the comment docs.

codecov · 2019-03-12T11:14:33Z

Codecov Report

Merging #25686 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #25686      +/-   ##
==========================================
+ Coverage   91.47%   91.48%   +<.01%     
==========================================
  Files         173      173              
  Lines       52872    52881       +9     
==========================================
+ Hits        48366    48376      +10     
+ Misses       4506     4505       -1

Flag	Coverage Δ
#multiple	`90.04% <100%> (ø)`	⬆️
#single	`41.81% <20%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/parsers.py	`95.67% <100%> (+0.08%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9821b77...bdea635. Read the comment docs.

doc/source/whatsnew/v0.25.0.rst

TomAugspurger · 2019-03-12T20:39:06Z

doc/source/whatsnew/v0.24.2.rst

@@ -18,7 +18,7 @@ including other versions of pandas.
 .. _whatsnew_0242.regressions:

 Fixed Regressions
-^^^^^^^^^^^^^^^^^


Looks like some git issues. Try reverting the changes to this file.

…25623

…ecols-out-of-bounds

…25623

gfyoung

Nice work, @heckeop !

@jreback back over to you

jreback · 2019-03-18T12:30:16Z

lgtm. @heckeop can you merge master. ping on green.

simonjayhawkins · 2019-03-18T15:20:08Z

pandas/tests/io/parser/test_usecols.py

+    data = "a,b,c\n1,2,3\n4,5,6"
+    parser = all_parsers
+
+    with pytest.raises(ValueError, match=_msg_validate_usecols_names):


although this technically doesn't fail the regex, it appears that _msg_validate_usecols_names is intended to be used with .format()

Yea good catch @simonjayhawkins

indeed! @jreback I'll push an update this weekend

WillAyd · 2019-03-19T04:40:36Z

pandas/tests/io/parser/test_usecols.py

+    data = "a,b,c\n1,2,3\n4,5,6"
+    parser = all_parsers
+
+    with pytest.raises(ValueError, match=_msg_validate_usecols_names):


Yea good catch @simonjayhawkins

jreback · 2019-03-19T23:30:21Z

can you merge master. ping on green.

jreback

pls merge master and update the whatsnew. ping on green.

jreback · 2019-03-22T11:37:47Z

doc/source/whatsnew/v0.25.0.rst

@@ -123,7 +123,7 @@ Bug Fixes
 ~~~~~~~~~
 - Bug in :func:`to_datetime` which would raise an (incorrect) ``ValueError`` when called with a date far into the future and the ``format`` argument specified instead of raising ``OutOfBoundsDatetime`` (:issue:`23830`)
 - Bug in an error message in :meth:`DataFrame.plot`. Improved the error message if non-numerics are passed to :meth:`DataFrame.plot` (:issue:`25481`)
-
+- Bug in ``read_csv`` which would not raise ``ValueError`` if a column index in ``usecols`` was out of bounds (:issue:`25623`)


this needs to be moved to the I/O section of bug fixes

…unds

…sing columns.

…unds

pep8speaks · 2019-03-23T21:03:34Z

Hello @heckeop! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-03-25 02:29:50 UTC

ghost · 2019-03-23T22:42:05Z

@jreback @WillAyd thoughts?

jreback

doc changes, ping on green.

jreback · 2019-03-24T15:37:35Z

doc/source/whatsnew/v0.25.0.rst

@@ -182,7 +182,8 @@ Performance Improvements

 Bug Fixes
 ~~~~~~~~~
-
+- Bug in :func:`to_datetime` which would raise an (incorrect) ``ValueError`` when called with a date far into the future and the ``format`` argument specified instead of raising ``OutOfBoundsDatetime`` (:issue:`23830`)


remove these; these have already been moved

pandas/tests/io/parser/test_usecols.py

pandas/io/parsers.py

jreback · 2019-03-24T15:39:13Z

pandas/io/parsers.py

@@ -2632,6 +2644,10 @@ def _infer_columns(self):
            ncols = len(line)
            num_original_columns = ncols

+            # GH25623


here as well

…bing the validations.

…unds

pandas/tests/io/parser/test_usecols.py

ghost · 2019-03-30T14:33:13Z

@jreback Thoughts?

jreback · 2019-03-30T17:36:36Z

thanks @heckeop the test is pretty clear i think.

ghost mentioned this pull request Mar 12, 2019

BUG: Raise ValueError if a column index in usecols is out of bounds. #25685

Closed

TomAugspurger reviewed Mar 12, 2019

View reviewed changes

sk-tests added 5 commits March 12, 2019 18:06

BUG: Raise ValueError if a column index in usecols is out of bounds. #…

caec3c5

…25623

BUG: Raise ValueError if a column index in usecols is out of bounds. #…

10126e1

…25623

update v0.24.2.rst

4eb55be

Merge branch 'master' of https://github.com/pandas-dev/pandas into us…

c86114b

…ecols-out-of-bounds

BUG: Raise ValueError if a column index in usecols is out of bounds. #…

52d9a2d

…25623

jreback requested a review from gfyoung March 13, 2019 17:12

jreback added IO CSV read_csv, to_csv Error Reporting Incorrect or improved errors from pandas labels Mar 13, 2019

gfyoung approved these changes Mar 13, 2019

View reviewed changes

jreback added this to the 0.25.0 milestone Mar 18, 2019

simonjayhawkins reviewed Mar 18, 2019

View reviewed changes

WillAyd requested changes Mar 19, 2019

View reviewed changes

jreback requested changes Mar 22, 2019

View reviewed changes

sk-tests added 3 commits March 23, 2019 12:43

Merge remote-tracking branch 'upstream/master' into usecols-out-of-bo…

e04654b

…unds

use string fromatting with _msg_validate_usecols_names to specify mis…

2d22a93

…sing columns.

Merge remote-tracking branch 'upstream/master' into usecols-out-of-bo…

98193d5

…unds

Move the bug fix documentation to the I/O section

2d8835d

jreback requested changes Mar 24, 2019

View reviewed changes

sk-tests added 2 commits March 24, 2019 19:28

Add issue number as a comment in the test. Add 1-line comments descri…

048094d

…bing the validations.

Merge remote-tracking branch 'upstream/master' into usecols-out-of-bo…

bdea635

…unds

WillAyd reviewed Mar 28, 2019

View reviewed changes

pandas/tests/io/parser/test_usecols.py Show resolved Hide resolved

jreback approved these changes Mar 30, 2019

View reviewed changes

jreback merged commit 025b5dc into pandas-dev:master Mar 30, 2019

Uh oh!

BUG: Raise ValueError if a column index in usecols is out of bounds. … #25686

BUG: Raise ValueError if a column index in usecols is out of bounds. … #25686

Uh oh!

Conversation

ghost commented Mar 12, 2019

Uh oh!

codecov bot commented Mar 12, 2019

Codecov Report

Uh oh!

codecov bot commented Mar 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gfyoung left a comment

Choose a reason for hiding this comment

Uh oh!

jreback commented Mar 18, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jreback commented Mar 19, 2019

Uh oh!

jreback left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pep8speaks commented Mar 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated at 2019-03-25 02:29:50 UTC

Uh oh!

ghost commented Mar 23, 2019

Uh oh!

jreback left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ghost commented Mar 30, 2019 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jreback commented Mar 30, 2019

Uh oh!

Uh oh!

codecov bot commented Mar 12, 2019 •

edited

Loading

pep8speaks commented Mar 23, 2019 •

edited

Loading

ghost commented Mar 30, 2019 •

edited by ghost

Loading