BUG: More followups on to_datetime exceptions, xref #13033

Author: Jeff Reback <[email protected]>

Closes #13059 from jreback/to_datetime3 and squashes the following commits:

6cd8e0f [Jeff Reback] BUG: More followups on to_datetime exceptions, xref #13033

Author: jreback

doc/source/whatsnew/v0.18.1.txt

@@ -11,9 +11,9 @@ Highlights include:
 
 - ``.groupby(...)`` has been enhanced to provide convenient syntax when working with ``.rolling(..)``, ``.expanding(..)`` and ``.resample(..)`` per group, see :ref:`here <whatsnew_0181.deferred_ops>`
 - ``pd.to_datetime()`` has gained the ability to assemble dates from a ``DataFrame``, see :ref:`here <whatsnew_0181.enhancements.assembling>`
+- Method chaining improvements, see :ref:`here <whatsnew_0181.enhancements.method_chain>`.
 - Custom business hour offset, see :ref:`here <whatsnew_0181.enhancements.custombusinesshour>`.
 - Many bug fixes in the handling of ``sparse``, see :ref:`here <whatsnew_0181.sparse>`
-- Method chaining improvements, see :ref:`here <whatsnew_0181.enhancements.method_chain>`.
 - Expanded the :ref:`Tutorials section <tutorial-modern>` with a feature on modern pandas, courtesy of `@TomAugsburger <https://twitter.com/TomAugspurger>`__. (:issue:`13045`).
 
 
@@ -40,12 +40,19 @@ see :ref:`Custom Business Hour <timeseries.custombusinesshour>` (:issue:`11514`)
     from pandas.tseries.offsets import CustomBusinessHour
     from pandas.tseries.holiday import USFederalHolidayCalendar
     bhour_us = CustomBusinessHour(calendar=USFederalHolidayCalendar())
-    # Friday before MLK Day
+
+Friday before MLK Day
+
+.. ipython:: python
+
     dt = datetime(2014, 1, 17, 15)
 
     dt + bhour_us
 
-    # Tuesday after MLK Day (Monday is skipped because it's a holiday)
+Tuesday after MLK Day (Monday is skipped because it's a holiday)
+
+.. ipython:: python
+
     dt + bhour_us * 2
 
 .. _whatsnew_0181.deferred_ops:
@@ -102,8 +109,8 @@ Now you can do:
 Method chaininng improvements
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The following methods / indexers now accept ``callable``. It is intended to make
-these more useful in method chains, see :ref:`Selection By Callable <indexing.callable>`.
+The following methods / indexers now accept a ``callable``. It is intended to make
+these more useful in method chains, see the :ref:`documentation <indexing.callable>`.
 (:issue:`11485`, :issue:`12533`)
 
 - ``.where()`` and ``.mask()``
@@ -113,7 +120,7 @@ these more useful in method chains, see :ref:`Selection By Callable <indexing.ca
 ``.where()`` and ``.mask()``
 """"""""""""""""""""""""""""
 
-These can accept a callable as condition and ``other``
+These can accept a callable for the condition and ``other``
 arguments.
 
 .. ipython:: python
@@ -126,8 +133,8 @@ arguments.
 ``.loc[]``, ``.iloc[]``, ``.ix[]``
 """"""""""""""""""""""""""""""""""
 
-These can accept a callable, and tuple of callable as a slicer. The callable
-can return valid ``bool`` indexer or anything which is valid for these indexer's input.
+These can accept a callable, and a tuple of callable as a slicer. The callable
+can return a valid boolean indexer or anything which is valid for these indexer's input.
 
 .. ipython:: python
 
@@ -141,7 +148,7 @@ can return valid ``bool`` indexer or anything which is valid for these indexer's
 """""""""""""""
 
 Finally, you can use a callable in ``[]`` indexing of Series, DataFrame and Panel.
-The callable must return valid input for ``[]`` indexing depending on its
+The callable must return a valid input for ``[]`` indexing depending on its
 class and index type.
 
 .. ipython:: python
@@ -154,8 +161,10 @@ without using temporary variable.
 .. ipython:: python
 
    bb = pd.read_csv('data/baseball.csv', index_col='id')
-   (bb.groupby(['year', 'team']).sum()
-      .loc[lambda df: df.r > 100])
+   (bb.groupby(['year', 'team'])
+      .sum()
+      .loc[lambda df: df.r > 100]
+   )
 
 .. _whatsnew_0181.partial_string_indexing:
 
@@ -174,8 +183,14 @@ Partial string indexing now matches on ``DateTimeIndex`` when part of a ``MultiI
                                                         ['a', 'b']]))
    dft2
    dft2.loc['2013-01-05']
+
+On other levels
+
+.. ipython:: python
+
    idx = pd.IndexSlice
    dft2 = dft2.swaplevel(0, 1).sort_index()
+   dft2
    dft2.loc[idx[:, '2013-01-05'], :]
 
 .. _whatsnew_0181.enhancements.assembling:
@@ -225,7 +240,9 @@ Other Enhancements
   .. ipython:: python
 
      idx = pd.Index([1., 2., 3., 4.], dtype='float')
-     idx.take([2, -1])     # default, allow_fill=True, fill_value=None
+
+     # default, allow_fill=True, fill_value=None
+     idx.take([2, -1])
      idx.take([2, -1], fill_value=True)
 
 - ``Index`` now supports ``.str.get_dummies()`` which returns ``MultiIndex``, see :ref:`Creating Indicator Variables <text.indicator>` (:issue:`10008`, :issue:`10103`)
@@ -362,7 +379,7 @@ New Behavior:
 numpy function compatibility
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Compatibility between pandas array-like methods (e.g. ```sum`` and ``take``) and their ``numpy``
+Compatibility between pandas array-like methods (e.g. ``sum`` and ``take``) and their ``numpy``
 counterparts has been greatly increased by augmenting the signatures of the ``pandas`` methods so
 as to accept arguments that can be passed in from ``numpy``, even if they are not necessarily
 used in the ``pandas`` implementation (:issue:`12644`, :issue:`12638`, :issue:`12687`)
@@ -436,12 +453,12 @@ New Behavior:
 Changes in ``read_csv`` exceptions
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-In order to standardize the ``read_csv`` API for both the C and Python engines, both will now raise an
+In order to standardize the ``read_csv`` API for both the ``c`` and ``python`` engines, both will now raise an
 ``EmptyDataError``, a subclass of ``ValueError``, in response to empty columns or header (:issue:`12493`, :issue:`12506`)
 
 Previous behaviour:
 
-.. code-block:: python
+.. code-block:: ipython
 
    In [1]: df = pd.read_csv(StringIO(''), engine='c')
    ...
@@ -453,7 +470,7 @@ Previous behaviour:
 
 New behaviour:
 
-.. code-block:: python
+.. code-block:: ipython
 
    In [1]: df = pd.read_csv(StringIO(''), engine='c')
    ...
@@ -465,10 +482,10 @@ New behaviour:
 
 In addition to this error change, several others have been made as well:
 
-- ``CParserError`` is now a ``ValueError`` instead of just an ``Exception`` (:issue:`12551`)
-- A ``CParserError`` is now raised instead of a generic ``Exception`` in ``read_csv`` when the C engine cannot parse a column (:issue:`12506`)
-- A ``ValueError`` is now raised instead of a generic ``Exception`` in ``read_csv`` when the C engine encounters a ``NaN`` value in an integer column (:issue:`12506`)
-- A ``ValueError`` is now raised instead of a generic ``Exception`` in ``read_csv`` when ``true_values`` is specified, and the C engine encounters an element in a column containing unencodable bytes (:issue:`12506`)
+- ``CParserError`` now sub-classes ``ValueError`` instead of just a ``Exception`` (:issue:`12551`)
+- A ``CParserError`` is now raised instead of a generic ``Exception`` in ``read_csv`` when the ``c`` engine cannot parse a column (:issue:`12506`)
+- A ``ValueError`` is now raised instead of a generic ``Exception`` in ``read_csv`` when the ``c`` engine encounters a ``NaN`` value in an integer column (:issue:`12506`)
+- A ``ValueError`` is now raised instead of a generic ``Exception`` in ``read_csv`` when ``true_values`` is specified, and the ``c`` engine encounters an element in a column containing unencodable bytes (:issue:`12506`)
 - ``pandas.parser.OverflowError`` exception has been removed and has been replaced with Python's built-in ``OverflowError`` exception (:issue:`12506`)
 - ``pd.read_csv()`` no longer allows a combination of strings and integers for the ``usecols`` parameter (:issue:`12678`)
 
@@ -478,24 +495,33 @@ In addition to this error change, several others have been made as well:
 ``to_datetime`` error changes
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Bugs in ``pd.to_datetime()`` when passing a ``unit`` with convertible entries and ``errors='coerce'`` or non-convertible with ``errors='ignore'`` (:issue:`11758`, :issue:`13052`)
+Bugs in ``pd.to_datetime()`` when passing a ``unit`` with convertible entries and ``errors='coerce'`` or non-convertible with ``errors='ignore'``. Furthermore, an ``OutOfBoundsDateime`` exception will be raised when an out-of-range value is encountered for that unit when ``errors='raise'``. (:issue:`11758`, :issue:`13052`, :issue:`13059`)
 
 Previous behaviour:
 
-.. code-block:: python
+.. code-block:: ipython
 
    In [27]: pd.to_datetime(1420043460, unit='s', errors='coerce')
    Out[27]: NaT
 
    In [28]: pd.to_datetime(11111111, unit='D', errors='ignore')
    OverflowError: Python int too large to convert to C long
 
+   In [29]: pd.to_datetime(11111111, unit='D', errors='raise')
+   OverflowError: Python int too large to convert to C long
+
 New behaviour:
 
-.. ipython:: python
+.. code-block:: ipython
+
+   In [2]: pd.to_datetime(1420043460, unit='s', errors='coerce')
+   Out[2]: Timestamp('2014-12-31 16:31:00')
+
+   In [3]: pd.to_datetime(11111111, unit='D', errors='ignore')
+   Out[3]: 11111111
 
-   pd.to_datetime(1420043460, unit='s', errors='coerce')
-   pd.to_datetime(11111111, unit='D', errors='ignore')
+   In [4]: pd.to_datetime(11111111, unit='D', errors='raise')
+   OutOfBoundsDatetime: cannot convert input with unit 'D'
 
 .. _whatsnew_0181.api.other:
 
@@ -505,14 +531,14 @@ Other API changes
 - ``.swaplevel()`` for ``Series``, ``DataFrame``, ``Panel``, and ``MultiIndex`` now features defaults for its first two parameters ``i`` and ``j`` that swap the two innermost levels of the index. (:issue:`12934`)
 - ``.searchsorted()`` for ``Index`` and ``TimedeltaIndex`` now accept a ``sorter`` argument to maintain compatibility with numpy's ``searchsorted`` function (:issue:`12238`)
 - ``Period`` and ``PeriodIndex`` now raises ``IncompatibleFrequency`` error which inherits ``ValueError`` rather than raw ``ValueError`` (:issue:`12615`)
-- ``Series.apply`` for category dtype now applies the passed function to each ``.categories`` (not ``.codes``), and returns a ``category`` dtype if possible (:issue:`12473`)
-- ``read_csv`` will now raise a ``TypeError`` if ``parse_dates`` is neither a boolean, list, or dictionary (:issue:`5636`)
-- The default for ``.query()/.eval()`` is now ``engine=None``, which will use ``numexpr`` if it's installed; otherwise it will fallback to the ``python`` engine. This mimics the pre-0.18.1 behavior if ``numexpr`` is installed (and which Previously, if numexpr was not installed, ``.query()/.eval()`` would raise). (:issue:`12749`)
+- ``Series.apply`` for category dtype now applies the passed function to each of the ``.categories`` (and not the ``.codes``), and returns a ``category`` dtype if possible (:issue:`12473`)
+- ``read_csv`` will now raise a ``TypeError`` if ``parse_dates`` is neither a boolean, list, or dictionary (matches the doc-string) (:issue:`5636`)
+- The default for ``.query()/.eval()`` is now ``engine=None``, which will use ``numexpr`` if it's installed; otherwise it will fallback to the ``python`` engine. This mimics the pre-0.18.1 behavior if ``numexpr`` is installed (and which, previously, if numexpr was not installed, ``.query()/.eval()`` would raise). (:issue:`12749`)
 - ``pd.show_versions()`` now includes ``pandas_datareader`` version (:issue:`12740`)
 - Provide a proper ``__name__`` and ``__qualname__`` attributes for generic functions (:issue:`12021`)
 - ``pd.concat(ignore_index=True)`` now uses ``RangeIndex`` as default (:issue:`12695`)
 - ``pd.merge()`` and ``DataFrame.join()`` will show a ``UserWarning`` when merging/joining a single- with a multi-leveled dataframe (:issue:`9455`, :issue:`12219`)
-- Compat with SciPy > 0.17 for deprecated ``piecewise_polynomial`` interpolation method (:issue:`12887`)
+- Compat with ``scipy`` > 0.17 for deprecated ``piecewise_polynomial`` interpolation method; support for the replacement ``from_derivatives`` method (:issue:`12887`)
 
 .. _whatsnew_0181.deprecations:
 
@@ -578,7 +604,8 @@ Bug Fixes
 - Bug in ``pd.crosstab()`` where would silently ignore ``aggfunc`` if ``values=None`` (:issue:`12569`).
 - Potential segfault in ``DataFrame.to_json`` when serialising ``datetime.time`` (:issue:`11473`).
 - Potential segfault in ``DataFrame.to_json`` when attempting to serialise 0d array (:issue:`11299`).
-- Segfault in ``to_json`` when attempting to serialise a ``DataFrame`` or ``Series`` with non-ndarray values (:issue:`10778`).
+- Segfault in ``to_json`` when attempting to serialise a ``DataFrame`` or ``Series`` with non-ndarray values; now supports serialization of ``category``, ``sparse``, and ``datetime64[ns, tz]`` dtypes (:issue:`10778`).
+- Bug in ``DataFrame.to_json`` with unsupported dtype not passed to default handler (:issue:`12554`).
 - Bug in ``.align`` not returning the sub-class (:issue:`12983`)
 - Bug in aligning a ``Series`` with a ``DataFrame`` (:issue:`13037`)
 - Bug in ``ABCPanel`` in which ``Panel4D`` was not being considered as a valid instance of this generic type (:issue:`12810`)
@@ -587,33 +614,32 @@ Bug Fixes
 - Bug in consistency of ``.name`` on ``.groupby(..).apply(..)`` cases (:issue:`12363`)
 
 - Bug in ``Timestamp.__repr__`` that caused ``pprint`` to fail in nested structures (:issue:`12622`)
-- Bug in ``Timedelta.min`` and ``Timedelta.max``, the properties now report the true minimum/maximum ``timedeltas`` as recognized by Pandas. See :ref:`documentation <timedeltas.limitations>`. (:issue:`12727`)
+- Bug in ``Timedelta.min`` and ``Timedelta.max``, the properties now report the true minimum/maximum ``timedeltas`` as recognized by pandas. See the :ref:`documentation <timedeltas.limitations>`. (:issue:`12727`)
 - Bug in ``.quantile()`` with interpolation may coerce to ``float`` unexpectedly (:issue:`12772`)
-- Bug in ``.quantile()`` with empty Series may return scalar rather than empty Series (:issue:`12772`)
+- Bug in ``.quantile()`` with empty ``Series`` may return scalar rather than empty ``Series`` (:issue:`12772`)
 
 
 - Bug in ``.loc`` with out-of-bounds in a large indexer would raise ``IndexError`` rather than ``KeyError`` (:issue:`12527`)
 - Bug in resampling when using a ``TimedeltaIndex`` and ``.asfreq()``, would previously not include the final fencepost (:issue:`12926`)
-- Bug in ``DataFrame.to_json`` with unsupported `dtype` not passed to default handler (:issue:`12554`).
 
 - Bug in equality testing with a ``Categorical`` in a ``DataFrame`` (:issue:`12564`)
 - Bug in ``GroupBy.first()``, ``.last()`` returns incorrect row when ``TimeGrouper`` is used (:issue:`7453`)
 
 
 
-- Bug in ``read_csv`` with the C engine when specifying ``skiprows`` with newlines in quoted items (:issue:`10911`, :issue:`12775`)
+- Bug in ``pd.read_csv()`` with the ``c`` engine when specifying ``skiprows`` with newlines in quoted items (:issue:`10911`, :issue:`12775`)
 - Bug in ``DataFrame`` timezone lost when assigning tz-aware datetime ``Series`` with alignment (:issue:`12981`)
 
 
 
 
-- Bug in ``value_counts`` when ``normalize=True`` and ``dropna=True`` where nulls still contributed to the normalized count (:issue:`12558`)
-- Bug in ``Series.value_counts()`` loses name if its dtype is category (:issue:`12835`)
+- Bug in ``.value_counts()`` when ``normalize=True`` and ``dropna=True`` where nulls still contributed to the normalized count (:issue:`12558`)
+- Bug in ``Series.value_counts()`` loses name if its dtype is ``category`` (:issue:`12835`)
 - Bug in ``Series.value_counts()`` loses timezone info (:issue:`12835`)
 - Bug in ``Series.value_counts(normalize=True)`` with ``Categorical`` raises ``UnboundLocalError`` (:issue:`12835`)
 - Bug in ``Panel.fillna()`` ignoring ``inplace=True`` (:issue:`12633`)
-- Bug in ``read_csv`` when specifying ``names``, ``usecols``, and ``parse_dates`` simultaneously with the C engine (:issue:`9755`)
-- Bug in ``read_csv`` when specifying ``delim_whitespace=True`` and ``lineterminator`` simultaneously with the C engine (:issue:`12912`)
+- Bug in ``pd.read_csv()`` when specifying ``names``, ``usecols``, and ``parse_dates`` simultaneously with the ``c`` engine (:issue:`9755`)
+- Bug in ``pd.read_csv()`` when specifying ``delim_whitespace=True`` and ``lineterminator`` simultaneously with the ``c`` engine (:issue:`12912`)
 - Bug in ``Series.rename``, ``DataFrame.rename`` and ``DataFrame.rename_axis`` not treating ``Series`` as mappings to relabel (:issue:`12623`).
 - Clean in ``.rolling.min`` and ``.rolling.max`` to enhance dtype handling (:issue:`12373`)
 - Bug in ``groupby`` where complex types are coerced to float (:issue:`12902`)
@@ -635,25 +661,25 @@ Bug Fixes
 
 
 
-- Bug in ``concat`` raises ``AttributeError`` when input data contains tz-aware datetime and timedelta (:issue:`12620`)
-- Bug in ``concat`` did not handle empty ``Series`` properly (:issue:`11082`)
+- Bug in ``pd.concat`` raises ``AttributeError`` when input data contains tz-aware datetime and timedelta (:issue:`12620`)
+- Bug in ``pd.concat`` did not handle empty ``Series`` properly (:issue:`11082`)
 
 - Bug in ``.plot.bar`` alginment when ``width`` is specified with ``int`` (:issue:`12979`)
 
 
 - Bug in ``fill_value`` is ignored if the argument to a binary operator is a constant (:issue:`12723`)
 
-- Bug in ``pd.read_html`` when using bs4 flavor and parsing table with a header and only one column (:issue:`9178`)
+- Bug in ``pd.read_html()`` when using bs4 flavor and parsing table with a header and only one column (:issue:`9178`)
 
-- Bug in ``pivot_table`` when ``margins=True`` and ``dropna=True`` where nulls still contributed to margin count (:issue:`12577`)
-- Bug in ``pivot_table`` when ``dropna=False`` where table index/column names disappear (:issue:`12133`)
-- Bug in ``crosstab`` when ``margins=True`` and ``dropna=False`` which raised (:issue:`12642`)
+- Bug in ``.pivot_table`` when ``margins=True`` and ``dropna=True`` where nulls still contributed to margin count (:issue:`12577`)
+- Bug in ``.pivot_table`` when ``dropna=False`` where table index/column names disappear (:issue:`12133`)
+- Bug in ``pd.crosstab()`` when ``margins=True`` and ``dropna=False`` which raised (:issue:`12642`)
 
 - Bug in ``Series.name`` when ``name`` attribute can be a hashable type (:issue:`12610`)
 
 - Bug in ``.describe()`` resets categorical columns information (:issue:`11558`)
 - Bug where ``loffset`` argument was not applied when calling ``resample().count()`` on a timeseries (:issue:`12725`)
 - ``pd.read_excel()`` now accepts column names associated with keyword argument ``names`` (:issue:`12870`)
-- Bug in ``to_numeric`` with ``Index`` returns ``np.ndarray``, rather than ``Index`` (:issue:`12777`)
-- Bug in ``to_numeric`` with datetime-like may raise ``TypeError`` (:issue:`12777`)
-- Bug in ``to_numeric`` with scalar raises ``ValueError`` (:issue:`12777`)
+- Bug in ``pd.to_numeric()`` with ``Index`` returns ``np.ndarray``, rather than ``Index`` (:issue:`12777`)
+- Bug in ``pd.to_numeric()`` with datetime-like may raise ``TypeError`` (:issue:`12777`)
+- Bug in ``pd.to_numeric()`` with scalar raises ``ValueError`` (:issue:`12777`)