From 0b162f5d75b42b0fd581eac0f876f927bec7a681 Mon Sep 17 00:00:00 2001 From: Terji Petersen Date: Sun, 2 Apr 2023 10:55:15 +0100 Subject: [PATCH 1/6] DOC: cleanup whatsnew entry for na_action='ignore' --- doc/source/whatsnew/v2.1.0.rst | 51 ++++++++++++++++++++++++++++++---- 1 file changed, 45 insertions(+), 6 deletions(-) diff --git a/doc/source/whatsnew/v2.1.0.rst b/doc/source/whatsnew/v2.1.0.rst index f9f1b87ddaef8..ec7e77ae08f4b 100644 --- a/doc/source/whatsnew/v2.1.0.rst +++ b/doc/source/whatsnew/v2.1.0.rst @@ -21,8 +21,47 @@ enhancement1 .. _whatsnew_210.enhancements.enhancement2: -enhancement2 -^^^^^^^^^^^^ +``map(func, na_action="ignore")`` now works for all array types +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When given a callable, :meth:`Series.map` applies the callable to all elements of the ``Series``. +Similarly, :meth:`DataFrame.applymap` applies the callable to all elements of the ``DataFrame``, +while :meth:`Index.map` applies the callable to all elements of the ``Index``. + +Frequently, it is not desirable to apply the callable to nan-like values of the array and to avoid doing +that, the ``map`` method could be called with ``na_action="ignore"``, e.g. like this: ``ser.map(func, na_action="ignore")``. +However, ``na_action="ignore"`` was not implemented for many ``ExtensionArray`` and ``Index`` types +and whether ``na_action="ignore"`` worked with a specific data type was quite unpredictable. + +``na_action="ignore"`` now works for all array types (:issue:`52219`, :issue:`51645`, :issue:`51809`, :issue:`51936`, :issue:`52033`; :issue:`52096`). + +*Previous behavior*: + +.. code-block:: ipython + + In [1]: ser = pd.Series(["a", "b", np.nan], dtype="category") + In [2]: ser.map(str.upper, na_action="ignore") + NotImplementedError + In [3]: df = pd.DataFrame(ser) + In [4]: df.applymap(str.upper, na_action="ignore") # worked for DataFrame + 0 + 0 A + 1 B + 2 NaN + In [5]: idx = pd.Index(ser) + In [6]: idx.map(str.upper, na_action="ignore") + TypeError: CategoricalIndex.map() got an unexpected keyword argument 'na_action' + +*New behavior*: + +.. ipython:: python + + ser = pd.Series(["a", "b", np.nan], dtype="category") + ser.map(str.upper, na_action="ignore") + df = pd.DataFrame(ser) + df.applymap(str.upper, na_action="ignore") + idx = pd.Index(ser) + idx.map(str.upper, na_action="ignore") .. _whatsnew_210.enhancements.other: @@ -34,6 +73,7 @@ Other enhancements - Implemented ``__pandas_priority__`` to allow custom types to take precedence over :class:`DataFrame`, :class:`Series`, :class:`Index`, or :class:`ExtensionArray` for arithmetic operations, :ref:`see the developer guide ` (:issue:`48347`) - :meth:`MultiIndex.sort_values` now supports ``na_position`` (:issue:`51612`) - :meth:`MultiIndex.sortlevel` and :meth:`Index.sortlevel` gained a new keyword ``na_position`` (:issue:`51612`) +- :meth:`arrays.DatetimeArray.map`, :meth:`arrays.TimedeltaArray.map` and :meth:`arrays.PeriodArray.map` can now take a ``na_action`` argument (:issue:`51644`) - Improve error message when setting :class:`DataFrame` with wrong number of columns through :meth:`DataFrame.isetitem` (:issue:`51701`) - Let :meth:`DataFrame.to_feather` accept a non-default :class:`Index` and non-string column names (:issue:`51787`) - :class:`api.extensions.ExtensionArray` now has a :meth:`~api.extensions.ExtensionArray.map` method (:issue:`51809`) @@ -42,7 +82,6 @@ Other enhancements - Improved error message when creating a DataFrame with empty data (0 rows), no index and an incorrect number of columns. (:issue:`52084`) - :meth:`DataFrame.applymap` now uses the :meth:`~api.extensions.ExtensionArray.map` method of underlying :class:`api.extensions.ExtensionArray` instances (:issue:`52219`) - :meth:`arrays.SparseArray.map` now supports ``na_action`` (:issue:`52096`). -- :meth:`Categorical.map` and :meth:`CategoricalIndex.map` now have a ``na_action`` parameter (:issue:`44279`) - Add dtype of categories to ``repr`` information of :class:`CategoricalDtype` (:issue:`52179`) - @@ -163,7 +202,7 @@ Categorical Datetimelike ^^^^^^^^^^^^ - Bug in :meth:`Timestamp.round` with values close to the implementation bounds returning incorrect results instead of raising ``OutOfBoundsDatetime`` (:issue:`51494`) -- :meth:`arrays.DatetimeArray.map` can now take a ``na_action`` argument. :meth:`DatetimeIndex.map` with ``na_action="ignore"`` now works as expected. (:issue:`51644`) +- :meth:`DatetimeIndex.map` with ``na_action="ignore"`` now works as expected. (:issue:`51644`) - Bug in :meth:`arrays.DatetimeArray.map` and :meth:`DatetimeIndex.map`, where the supplied callable operated array-wise instead of element-wise (:issue:`51977`) - @@ -171,7 +210,7 @@ Timedelta ^^^^^^^^^ - Bug in :meth:`Timedelta.round` with values close to the implementation bounds returning incorrect results instead of raising ``OutOfBoundsTimedelta`` (:issue:`51494`) - Bug in :class:`TimedeltaIndex` division or multiplication leading to ``.freq`` of "0 Days" instead of ``None`` (:issue:`51575`) -- :meth:`arrays.TimedeltaArray.map` can now take a ``na_action`` argument. :meth:`TimedeltaIndex.map` with ``na_action="ignore"`` now works as expected. (:issue:`51644`) +- :meth:`TimedeltaIndex.map` with ``na_action="ignore"`` now works as expected (:issue:`51644`) - Bug in :meth:`arrays.TimedeltaArray.map` and :meth:`TimedeltaIndex.map`, where the supplied callable operated array-wise instead of element-wise (:issue:`51977`) - @@ -227,7 +266,7 @@ I/O Period ^^^^^^ - Bug in :class:`PeriodDtype` constructor failing to raise ``TypeError`` when no argument is passed or when ``None`` is passed (:issue:`27388`) -- :meth:`arrays.PeriodArray.map` can now take a ``na_action`` argument. :meth:`PeriodIndex.map` with ``na_action="ignore"`` now works as expected. (:issue:`51644`) +- :meth:`PeriodIndex.map` with ``na_action="ignore"`` now works as expected (:issue:`51644`) - Bug in :class:`PeriodDtype` constructor raising ``ValueError`` instead of ``TypeError`` when an invalid type is passed (:issue:`51790`) - Bug in :meth:`arrays.PeriodArray.map` and :meth:`PeriodIndex.map`, where the supplied callable operated array-wise instead of element-wise (:issue:`51977`) - From ed5347352ff59757c6fa7a1c0e00465389ac1c56 Mon Sep 17 00:00:00 2001 From: Terji Petersen Date: Mon, 3 Apr 2023 22:28:39 +0100 Subject: [PATCH 2/6] Update doc/source/whatsnew/v2.1.0.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> --- doc/source/whatsnew/v2.1.0.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/whatsnew/v2.1.0.rst b/doc/source/whatsnew/v2.1.0.rst index ec7e77ae08f4b..60df1e6d08522 100644 --- a/doc/source/whatsnew/v2.1.0.rst +++ b/doc/source/whatsnew/v2.1.0.rst @@ -24,7 +24,7 @@ enhancement1 ``map(func, na_action="ignore")`` now works for all array types ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -When given a callable, :meth:`Series.map` applies the callable to all elements of the ``Series``. +When given a callable, :meth:`Series.map` applies the callable to all elements of the :class:`Series`. Similarly, :meth:`DataFrame.applymap` applies the callable to all elements of the ``DataFrame``, while :meth:`Index.map` applies the callable to all elements of the ``Index``. From acf7636c0ca65afa566cb50b107e81a798a8d511 Mon Sep 17 00:00:00 2001 From: Terji Petersen Date: Mon, 3 Apr 2023 22:28:48 +0100 Subject: [PATCH 3/6] Update doc/source/whatsnew/v2.1.0.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> --- doc/source/whatsnew/v2.1.0.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/whatsnew/v2.1.0.rst b/doc/source/whatsnew/v2.1.0.rst index 60df1e6d08522..8f1330d080244 100644 --- a/doc/source/whatsnew/v2.1.0.rst +++ b/doc/source/whatsnew/v2.1.0.rst @@ -25,7 +25,7 @@ enhancement1 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ When given a callable, :meth:`Series.map` applies the callable to all elements of the :class:`Series`. -Similarly, :meth:`DataFrame.applymap` applies the callable to all elements of the ``DataFrame``, +Similarly, :meth:`DataFrame.applymap` applies the callable to all elements of the :class:`DataFrame`, while :meth:`Index.map` applies the callable to all elements of the ``Index``. Frequently, it is not desirable to apply the callable to nan-like values of the array and to avoid doing From 3710f377e23fc91bc3c20a9fc0b3dd1154936b64 Mon Sep 17 00:00:00 2001 From: Terji Petersen Date: Mon, 3 Apr 2023 22:28:59 +0100 Subject: [PATCH 4/6] Update doc/source/whatsnew/v2.1.0.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> --- doc/source/whatsnew/v2.1.0.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/whatsnew/v2.1.0.rst b/doc/source/whatsnew/v2.1.0.rst index 8f1330d080244..88f44bf33ed71 100644 --- a/doc/source/whatsnew/v2.1.0.rst +++ b/doc/source/whatsnew/v2.1.0.rst @@ -26,7 +26,7 @@ enhancement1 When given a callable, :meth:`Series.map` applies the callable to all elements of the :class:`Series`. Similarly, :meth:`DataFrame.applymap` applies the callable to all elements of the :class:`DataFrame`, -while :meth:`Index.map` applies the callable to all elements of the ``Index``. +while :meth:`Index.map` applies the callable to all elements of the :class:`Index`. Frequently, it is not desirable to apply the callable to nan-like values of the array and to avoid doing that, the ``map`` method could be called with ``na_action="ignore"``, e.g. like this: ``ser.map(func, na_action="ignore")``. From 305dd02a48a59d4274410296f77c82c88bfdecf3 Mon Sep 17 00:00:00 2001 From: Terji Petersen Date: Mon, 3 Apr 2023 22:29:23 +0100 Subject: [PATCH 5/6] Update doc/source/whatsnew/v2.1.0.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> --- doc/source/whatsnew/v2.1.0.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/whatsnew/v2.1.0.rst b/doc/source/whatsnew/v2.1.0.rst index 88f44bf33ed71..9625512579a17 100644 --- a/doc/source/whatsnew/v2.1.0.rst +++ b/doc/source/whatsnew/v2.1.0.rst @@ -29,7 +29,7 @@ Similarly, :meth:`DataFrame.applymap` applies the callable to all elements of th while :meth:`Index.map` applies the callable to all elements of the :class:`Index`. Frequently, it is not desirable to apply the callable to nan-like values of the array and to avoid doing -that, the ``map`` method could be called with ``na_action="ignore"``, e.g. like this: ``ser.map(func, na_action="ignore")``. +that, the ``map`` method could be called with ``na_action="ignore"``, i.e. ``ser.map(func, na_action="ignore")``. However, ``na_action="ignore"`` was not implemented for many ``ExtensionArray`` and ``Index`` types and whether ``na_action="ignore"`` worked with a specific data type was quite unpredictable. From 8d50b8f99ccbfe34a3eeefb0971ecccc41a60662 Mon Sep 17 00:00:00 2001 From: Terji Petersen Date: Mon, 3 Apr 2023 23:08:33 +0100 Subject: [PATCH 6/6] update comment --- doc/source/whatsnew/v2.1.0.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/source/whatsnew/v2.1.0.rst b/doc/source/whatsnew/v2.1.0.rst index 9625512579a17..f1a713e5aebb2 100644 --- a/doc/source/whatsnew/v2.1.0.rst +++ b/doc/source/whatsnew/v2.1.0.rst @@ -31,7 +31,7 @@ while :meth:`Index.map` applies the callable to all elements of the :class:`Inde Frequently, it is not desirable to apply the callable to nan-like values of the array and to avoid doing that, the ``map`` method could be called with ``na_action="ignore"``, i.e. ``ser.map(func, na_action="ignore")``. However, ``na_action="ignore"`` was not implemented for many ``ExtensionArray`` and ``Index`` types -and whether ``na_action="ignore"`` worked with a specific data type was quite unpredictable. +and ``na_action="ignore"`` did not work correctly for any ``ExtensionArray`` subclass except the nullable numeric ones (i.e. with dtype :class:`Int64` etc.). ``na_action="ignore"`` now works for all array types (:issue:`52219`, :issue:`51645`, :issue:`51809`, :issue:`51936`, :issue:`52033`; :issue:`52096`). @@ -63,6 +63,10 @@ and whether ``na_action="ignore"`` worked with a specific data type was quite un idx = pd.Index(ser) idx.map(str.upper, na_action="ignore") +Also, note that :meth:`Categorical.map` implicitly has had its ``na_action`` set to ``"ignore"`` by default. +This has been deprecated and will :meth:`Categorical.map` in the future change the default +to ``na_action=None``, like for all the other array types. + .. _whatsnew_210.enhancements.other: Other enhancements