Skip to content

Fix pandas.Timedelta range #12728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 0 additions & 21 deletions doc/source/gotchas.rst
Original file line number Diff line number Diff line change
Expand Up @@ -356,27 +356,6 @@ such as ``numpy.logical_and``.
See the `this old issue <https://github.com/pydata/pandas/issues/2388>`__ for a more
detailed discussion.

.. _gotchas.timestamp-limits:

Timestamp limitations
---------------------

Minimum and maximum timestamps
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Since pandas represents timestamps in nanosecond resolution, the timespan that
can be represented using a 64-bit integer is limited to approximately 584 years:

.. ipython:: python

begin = pd.Timestamp.min
begin

end = pd.Timestamp.max
end

See :ref:`here <timeseries.oob>` for ways to represent data outside these bound.

Parsing Dates from Text Files
-----------------------------

Expand Down
20 changes: 20 additions & 0 deletions doc/source/timedeltas.rst
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,26 @@ The ``unit`` keyword argument specifies the unit of the Timedelta:
to_timedelta(np.arange(5), unit='s')
to_timedelta(np.arange(5), unit='d')

.. _timedeltas.limitations:

Timedelta limitations
~~~~~~~~~~~~~~~~~~~~~

Pandas represents ``Timedeltas`` in nanosecond resolution using
64 bit integers. As such, the 64 bit integer limits determine
the ``Timedelta`` limits.

.. ipython:: python
min_int = np.iinfo(np.int64).min
max_int = np.iinfo(np.int64).max

# Note: the smallest integer gives a NaT
Timedelta(min_int)
Timedelta(min_int+1) == Timedelta.min
Timedelta(max_int) == Timedelta.max

# (min_int - 1) and (max_int + 1) result in OverflowErrors

.. _timedeltas.operations:

Operations
Expand Down
23 changes: 22 additions & 1 deletion doc/source/timeseries.rst
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,27 @@ using various combinations of parameters like ``start``, ``end``,
The start and end dates are strictly inclusive. So it will not generate any
dates outside of those dates if specified.

.. _timeseries.timestamp-limits:

Timestamp limitations
---------------------

Minimum and maximum timestamps
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Since pandas represents timestamps in nanosecond resolution, the timespan that
can be represented using a 64-bit integer is limited to approximately 584 years:

.. ipython:: python

begin = pd.Timestamp.min
begin

end = pd.Timestamp.max
end

See :ref:`here <timeseries.oob>` for ways to represent data outside these bound.

.. _timeseries.datetimeindex:

DatetimeIndex
Expand Down Expand Up @@ -1691,7 +1712,7 @@ the quarter end:
Representing out-of-bounds spans
--------------------------------

If you have data that is outside of the ``Timestamp`` bounds, see :ref:`Timestamp limitations <gotchas.timestamp-limits>`,
If you have data that is outside of the ``Timestamp`` bounds, see :ref:`Timestamp limitations <timeseries.timestamp-limits>`,
then you can use a ``PeriodIndex`` and/or ``Series`` of ``Periods`` to do computations.

.. ipython:: python
Expand Down
1 change: 1 addition & 0 deletions doc/source/whatsnew/v0.18.1.txt
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ Bug Fixes


- Bug in ``Timestamp.__repr__`` that caused ``pprint`` to fail in nested structures (:issue:`12622`)
- Bug in ``Timedelta.min`` and ``Timedelta.max``, the properties now report the true minimum/maximum ``timedeltas`` as recognized by Pandas. See :ref:`documentation <timedeltas.limitations>`. (:issue:`12727`)



Expand Down
29 changes: 29 additions & 0 deletions pandas/tseries/tests/test_timedeltas.py
Original file line number Diff line number Diff line change
Expand Up @@ -1083,6 +1083,35 @@ def test_timedelta_hash_equality(self):
ns_td = Timedelta(1, 'ns')
self.assertNotEqual(hash(ns_td), hash(ns_td.to_pytimedelta()))

def test_implementation_limits(self):
min_td = Timedelta(Timedelta.min)
max_td = Timedelta(Timedelta.max)

# GH 12727
# timedelta limits correspond to int64 boundaries
self.assertTrue(min_td.value == np.iinfo(np.int64).min + 1)
self.assertTrue(max_td.value == np.iinfo(np.int64).max)

# Beyond lower limit, a NAT before the Overflow
self.assertIsInstance(min_td - Timedelta(1, 'ns'),
pd.tslib.NaTType)

with tm.assertRaises(OverflowError):
min_td - Timedelta(2, 'ns')

with tm.assertRaises(OverflowError):
max_td + Timedelta(1, 'ns')

# Same tests using the internal nanosecond values
td = Timedelta(min_td.value - 1, 'ns')
self.assertIsInstance(td, pd.tslib.NaTType)

with tm.assertRaises(OverflowError):
Timedelta(min_td.value - 2, 'ns')

with tm.assertRaises(OverflowError):
Timedelta(max_td.value + 1, 'ns')


class TestTimedeltaIndex(tm.TestCase):
_multiprocess_can_split_ = True
Expand Down
5 changes: 5 additions & 0 deletions pandas/tslib.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -2722,6 +2722,11 @@ class Timedelta(_Timedelta):
__pos__ = _op_unary_method(lambda x: x, '__pos__')
__abs__ = _op_unary_method(lambda x: abs(x), '__abs__')


# Resolution is in nanoseconds
Timedelta.min = Timedelta(np.iinfo(np.int64).min+1, 'ns')
Timedelta.max = Timedelta(np.iinfo(np.int64).max, 'ns')

cdef PyTypeObject* td_type = <PyTypeObject*> Timedelta

cdef inline bint is_timedelta(object o):
Expand Down