Skip to content

BUG: ignore errors for invalid dates in to_datetime() with errors=coerce (#25512) #26561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 1, 2019
Merged

BUG: ignore errors for invalid dates in to_datetime() with errors=coerce (#25512) #26561

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2d41f2f
BUG: ignore errors for invalid dates in to_datetime with coerce (#25512)
nathalier May 28, 2019
832b2ee
TST: add 2 tests for to_datetime() for Series with invalid values and…
nathalier May 29, 2019
e78b8a4
TST: create separate test test_to_datetime_format_YYYYMMDD_overflow()…
nathalier May 29, 2019
897dd8b
DOC: add whatsnew for #25512
nathalier May 29, 2019
e32641e
TST: parametrize test for GH #25512
nathalier May 30, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions pandas/core/tools/datetimes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -775,21 +775,21 @@ def calc_with_mask(carg, mask):
# try intlike / strings that are ints
try:
return calc(arg.astype(np.int64))
except ValueError:
except (ValueError, OverflowError):
pass

# a float with actual np.nan
try:
carg = arg.astype(np.float64)
return calc_with_mask(carg, notna(carg))
except ValueError:
except (ValueError, OverflowError):
pass

# string with NaN-like
try:
mask = ~algorithms.isin(arg, list(tslib.nat_strings))
return calc_with_mask(arg, mask)
except ValueError:
except (ValueError, OverflowError):
pass

return None
Expand Down
29 changes: 29 additions & 0 deletions pandas/tests/indexes/datetimes/test_tools.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,35 @@ def test_to_datetime_format_YYYYMMDD(self, cache):
result = to_datetime(s, format='%Y%m%d', cache=cache)
assert_series_equal(result, expected)

# GH 25512
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you make a new test (and _overflow) in the name

# NaN before strings with invalid date values, errors=coerce
s = Series(['19801222', np.nan, '20010012', '10019999'])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you parameterize over these Series inputs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it what you meant?
Thanks!

result = pd.to_datetime(s, format='%Y%m%d', errors='coerce',
cache=cache)
expected = Series([Timestamp('19801222'), np.nan, np.nan, np.nan])
tm.assert_series_equal(result, expected)

# NaN after strings with invalid date values, errors=coerce
s = Series(['19801222', '20010012', '10019999', np.nan])
result = pd.to_datetime(s, format='%Y%m%d', errors='coerce',
cache=cache)
expected = Series([Timestamp('19801222'), np.nan, np.nan, np.nan])
tm.assert_series_equal(result, expected)

# NaN before integers with invalid date values, errors=coerce
s = Series([20190813, np.nan, 20010012, 20019999])
result = pd.to_datetime(s, format='%Y%m%d', errors='coerce',
cache=cache)
expected = Series([Timestamp('20190813'), np.nan, np.nan, np.nan])
tm.assert_series_equal(result, expected)

# NaN after integers with invalid date values, errors=coerce
s = Series([20190813, 20010012, np.nan, 20019999])
result = pd.to_datetime(s, format='%Y%m%d', errors='coerce',
cache=cache)
expected = Series([Timestamp('20190813'), np.nan, np.nan, np.nan])
tm.assert_series_equal(result, expected)

# coercion
# GH 7930
s = Series([20121231, 20141231, 99991231])
Expand Down