API deprecate date_parser, add date_format (#51019)

* wip

* fixup

* update user guide

* whatsnew

* gh number

* update user guide;

* whatsnew

* update whatsnew note with date_format enhancement

* make example ipython code-block

* add tests for date_format

* wip add this to read_excel too

* validate within _parser

* minor fixup

* mention other readers in whatsnew, None -> no_default

* Update v2.0.0.rst

* fixup merge conflict resolution

---------

Co-authored-by: MarcoGorelli <>
Co-authored-by: Patrick Hoefler <[email protected]>

Author: MarcoGorelli

doc/source/user_guide/io.rst

@@ -290,6 +290,16 @@ date_parser : function, default ``None``
   values from the columns defined by parse_dates into a single array and pass
   that; and 3) call date_parser once for each row using one or more strings
   (corresponding to the columns defined by parse_dates) as arguments.
+
+  .. deprecated:: 2.0.0
+   Use ``date_format`` instead, or read in as ``object`` and then apply
+   :func:`to_datetime` as-needed.
+date_format : str, default ``None``
+   If used in conjunction with ``parse_dates``, will parse dates according to this
+   format. For anything more complex (e.g. different formats for different columns),
+   please read in as ``object`` and then apply :func:`to_datetime` as-needed.
+
+    .. versionadded:: 2.0.0
 dayfirst : boolean, default ``False``
   DD/MM format dates, international and European format.
 cache_dates : boolean, default True
@@ -800,7 +810,7 @@ Specifying date columns
 +++++++++++++++++++++++
 
 To better facilitate working with datetime data, :func:`read_csv`
-uses the keyword arguments ``parse_dates`` and ``date_parser``
+uses the keyword arguments ``parse_dates`` and ``date_format``
 to allow users to specify a variety of columns and date/time formats to turn the
 input text data into ``datetime`` objects.
 
@@ -898,33 +908,15 @@ data columns:
 Date parsing functions
 ++++++++++++++++++++++
 
-Finally, the parser allows you to specify a custom ``date_parser`` function to
-take full advantage of the flexibility of the date parsing API:
-
-.. ipython:: python
-
-   df = pd.read_csv(
-       "tmp.csv", header=None, parse_dates=date_spec, date_parser=pd.to_datetime
-   )
-   df
-
-pandas will try to call the ``date_parser`` function in three different ways. If
-an exception is raised, the next one is tried:
-
-1. ``date_parser`` is first called with one or more arrays as arguments,
-   as defined using ``parse_dates`` (e.g., ``date_parser(['2013', '2013'], ['1', '2'])``).
-
-2. If #1 fails, ``date_parser`` is called with all the columns
-   concatenated row-wise into a single array (e.g., ``date_parser(['2013 1', '2013 2'])``).
+Finally, the parser allows you to specify a custom ``date_format``.
+Performance-wise, you should try these methods of parsing dates in order:
 
-Note that performance-wise, you should try these methods of parsing dates in order:
+1. If you know the format, use ``date_format``, e.g.:
+   ``date_format="%d/%m/%Y"``.
 
-1. If you know the format, use ``pd.to_datetime()``:
-   ``date_parser=lambda x: pd.to_datetime(x, format=...)``.
-
-2. If you have a really non-standard format, use a custom ``date_parser`` function.
-   For optimal performance, this should be vectorized, i.e., it should accept arrays
-   as arguments.
+2. If you different formats for different columns, or want to pass any extra options (such
+   as ``utc``) to ``to_datetime``, then you should read in your data as ``object`` dtype, and
+   then use ``to_datetime``.
 
 
 .. ipython:: python
@@ -952,16 +944,13 @@ an object-dtype column with strings, even with ``parse_dates``.
    df = pd.read_csv(StringIO(content), parse_dates=["a"])
    df["a"]
 
-To parse the mixed-timezone values as a datetime column, pass a partially-applied
-:func:`to_datetime` with ``utc=True`` as the ``date_parser``.
+To parse the mixed-timezone values as a datetime column, read in as ``object`` dtype and
+then call :func:`to_datetime` with ``utc=True``.
 
 .. ipython:: python
 
-   df = pd.read_csv(
-       StringIO(content),
-       parse_dates=["a"],
-       date_parser=lambda col: pd.to_datetime(col, utc=True),
-   )
+   df = pd.read_csv(StringIO(content))
+   df["a"] = pd.to_datetime(df["a"], utc=True)
    df["a"]
 
 

doc/source/whatsnew/v0.24.0.rst

@@ -686,11 +686,19 @@ Parsing mixed-timezones with :func:`read_csv`
 As can be seen, the ``dtype`` is object; each value in the column is a string.
 To convert the strings to an array of datetimes, the ``date_parser`` argument
 
-.. ipython:: python
+.. code-block:: ipython
 
-   df = pd.read_csv(io.StringIO(content), parse_dates=['a'],
-                    date_parser=lambda col: pd.to_datetime(col, utc=True))
-   df.a
+   In [3]: df = pd.read_csv(
+      ...:     io.StringIO(content),
+      ...:     parse_dates=['a'],
+      ...:     date_parser=lambda col: pd.to_datetime(col, utc=True),
+      ...: )
+
+   In [4]: df.a
+   Out[4]:
+   0   1999-12-31 19:00:00+00:00
+   1   1999-12-31 18:00:00+00:00
+   Name: a, dtype: datetime64[ns, UTC]
 
 See :ref:`whatsnew_0240.api.timezone_offset_parsing` for more.
 

doc/source/whatsnew/v2.0.0.rst

@@ -316,6 +316,7 @@ Other enhancements
 - Added :meth:`DatetimeIndex.as_unit` and :meth:`TimedeltaIndex.as_unit` to convert to different resolutions; supported resolutions are "s", "ms", "us", and "ns" (:issue:`50616`)
 - Added :meth:`Series.dt.unit` and :meth:`Series.dt.as_unit` to convert to different resolutions; supported resolutions are "s", "ms", "us", and "ns" (:issue:`51223`)
 - Added new argument ``dtype`` to :func:`read_sql` to be consistent with :func:`read_sql_query` (:issue:`50797`)
+- :func:`read_csv`, :func:`read_table`, :func:`read_fwf` and :func:`read_excel` now accept ``date_format`` (:issue:`50601`)
 - :func:`to_datetime` now accepts ``"ISO8601"`` as an argument to ``format``, which will match any ISO8601 string (but possibly not identically-formatted) (:issue:`50411`)
 - :func:`to_datetime` now accepts ``"mixed"`` as an argument to ``format``, which will infer the format for each element individually (:issue:`50972`)
 - Added new argument ``engine`` to :func:`read_json` to support parsing JSON with pyarrow by specifying ``engine="pyarrow"`` (:issue:`48893`)
@@ -832,6 +833,7 @@ Deprecations
 - :meth:`Index.is_categorical` has been deprecated. Use :func:`pandas.api.types.is_categorical_dtype` instead (:issue:`50042`)
 - :meth:`Index.is_object` has been deprecated. Use :func:`pandas.api.types.is_object_dtype` instead (:issue:`50042`)
 - :meth:`Index.is_interval` has been deprecated. Use :func:`pandas.api.types.is_interval_dtype` instead (:issue:`50042`)
+- Deprecated argument ``date_parser`` in :func:`read_csv`, :func:`read_table`, :func:`read_fwf`, and :func:`read_excel` in favour of ``date_format`` (:issue:`50601`)
 - Deprecated ``all`` and ``any`` reductions with ``datetime64`` and :class:`DatetimeTZDtype` dtypes, use e.g. ``(obj != pd.Timestamp(0), tz=obj.tz).all()`` instead (:issue:`34479`)
 - Deprecated unused arguments ``*args`` and ``**kwargs`` in :class:`Resampler` (:issue:`50977`)
 - Deprecated calling ``float`` or ``int`` on a single element :class:`Series` to return a ``float`` or ``int`` respectively. Extract the element before calling ``float`` or ``int`` instead (:issue:`51101`)

pandas/io/excel/_base.py

@@ -250,6 +250,16 @@
     and pass that; and 3) call `date_parser` once for each row using one or
     more strings (corresponding to the columns defined by `parse_dates`) as
     arguments.
+
+  .. deprecated:: 2.0.0
+   Use ``date_format`` instead, or read in as ``object`` and then apply
+   :func:`to_datetime` as-needed.
+date_format : str, default ``None``
+   If used in conjunction with ``parse_dates``, will parse dates according to this
+   format. For anything more complex (e.g. different formats for different columns),
+   please read in as ``object`` and then apply :func:`to_datetime` as-needed.
+
+    .. versionadded:: 2.0.0
 thousands : str, default None
     Thousands separator for parsing string columns to numeric.  Note that
     this parameter is only necessary for columns stored as TEXT in Excel,
@@ -386,7 +396,8 @@ def read_excel(
     na_filter: bool = ...,
     verbose: bool = ...,
     parse_dates: list | dict | bool = ...,
-    date_parser: Callable | None = ...,
+    date_parser: Callable | lib.NoDefault = ...,
+    date_format: str | None = ...,
     thousands: str | None = ...,
     decimal: str = ...,
     comment: str | None = ...,
@@ -425,7 +436,8 @@ def read_excel(
     na_filter: bool = ...,
     verbose: bool = ...,
     parse_dates: list | dict | bool = ...,
-    date_parser: Callable | None = ...,
+    date_parser: Callable | lib.NoDefault = ...,
+    date_format: str | None = ...,
     thousands: str | None = ...,
     decimal: str = ...,
     comment: str | None = ...,
@@ -464,7 +476,8 @@ def read_excel(
     na_filter: bool = True,
     verbose: bool = False,
     parse_dates: list | dict | bool = False,
-    date_parser: Callable | None = None,
+    date_parser: Callable | lib.NoDefault = lib.no_default,
+    date_format: str | None = None,
     thousands: str | None = None,
     decimal: str = ".",
     comment: str | None = None,
@@ -508,6 +521,7 @@ def read_excel(
             verbose=verbose,
             parse_dates=parse_dates,
             date_parser=date_parser,
+            date_format=date_format,
             thousands=thousands,
             decimal=decimal,
             comment=comment,
@@ -711,7 +725,8 @@ def parse(
         na_values=None,
         verbose: bool = False,
         parse_dates: list | dict | bool = False,
-        date_parser: Callable | None = None,
+        date_parser: Callable | lib.NoDefault = lib.no_default,
+        date_format: str | None = None,
         thousands: str | None = None,
         decimal: str = ".",
         comment: str | None = None,
@@ -870,6 +885,7 @@ def parse(
                     skip_blank_lines=False,  # GH 39808
                     parse_dates=parse_dates,
                     date_parser=date_parser,
+                    date_format=date_format,
                     thousands=thousands,
                     decimal=decimal,
                     comment=comment,
@@ -1537,7 +1553,8 @@ def parse(
         nrows: int | None = None,
         na_values=None,
         parse_dates: list | dict | bool = False,
-        date_parser: Callable | None = None,
+        date_parser: Callable | lib.NoDefault = lib.no_default,
+        date_format: str | None = None,
         thousands: str | None = None,
         comment: str | None = None,
         skipfooter: int = 0,
@@ -1570,6 +1587,7 @@ def parse(
             na_values=na_values,
             parse_dates=parse_dates,
             date_parser=date_parser,
+            date_format=date_format,
             thousands=thousands,
             comment=comment,
             skipfooter=skipfooter,

pandas/io/parsers/base_parser.py

@@ -114,7 +114,8 @@ def __init__(self, kwds) -> None:
 
         self.parse_dates = _validate_parse_dates_arg(kwds.pop("parse_dates", False))
         self._parse_date_cols: Iterable = []
-        self.date_parser = kwds.pop("date_parser", None)
+        self.date_parser = kwds.pop("date_parser", lib.no_default)
+        self.date_format = kwds.pop("date_format", None)
         self.dayfirst = kwds.pop("dayfirst", False)
         self.keep_date_col = kwds.pop("keep_date_col", False)
 
@@ -133,6 +134,7 @@ def __init__(self, kwds) -> None:
 
         self._date_conv = _make_date_converter(
             date_parser=self.date_parser,
+            date_format=self.date_format,
             dayfirst=self.dayfirst,
             cache_dates=self.cache_dates,
         )
@@ -1089,16 +1091,30 @@ def _get_empty_meta(
 
 
 def _make_date_converter(
-    date_parser=None,
+    date_parser=lib.no_default,
     dayfirst: bool = False,
     cache_dates: bool = True,
+    date_format: str | None = None,
 ):
+    if date_parser is not lib.no_default:
+        warnings.warn(
+            "The argument 'date_parser' is deprecated and will "
+            "be removed in a future version. "
+            "Please use 'date_format' instead, or read your data in as 'object' dtype "
+            "and then call 'to_datetime'.",
+            FutureWarning,
+            stacklevel=find_stack_level(),
+        )
+    if date_parser is not lib.no_default and date_format is not None:
+        raise TypeError("Cannot use both 'date_parser' and 'date_format'")
+
     def converter(*date_cols):
-        if date_parser is None:
+        if date_parser is lib.no_default:
             strs = parsing.concat_date_cols(date_cols)
 
             return tools.to_datetime(
                 ensure_object(strs),
+                format=date_format,
                 utc=False,
                 dayfirst=dayfirst,
                 errors="ignore",
@@ -1152,7 +1168,8 @@ def converter(*date_cols):
     "parse_dates": False,
     "keep_date_col": False,
     "dayfirst": False,
-    "date_parser": None,
+    "date_parser": lib.no_default,
+    "date_format": None,
     "usecols": None,
     # 'iterator': False,
     "chunksize": None,