Merge remote-tracking branch 'upstream/master'

Author: jnecus

.travis.yml

@@ -75,8 +75,7 @@ matrix:
 
 before_install:
   - echo "before_install"
-  # set non-blocking IO on travis
-  # https://github.com/travis-ci/travis-ci/issues/8920#issuecomment-352661024
+  # Use blocking IO on travis.  Ref:  https://github.com/travis-ci/travis-ci/issues/8920#issuecomment-352661024
   - python -c 'import os,sys,fcntl; flags = fcntl.fcntl(sys.stdout, fcntl.F_GETFL); fcntl.fcntl(sys.stdout, fcntl.F_SETFL, flags&~os.O_NONBLOCK);'
   - source ci/travis_process_gbq_encryption.sh
   - export PATH="$HOME/miniconda3/bin:$PATH"

README.md

@@ -16,6 +16,7 @@
 [![Downloads](https://anaconda.org/conda-forge/pandas/badges/downloads.svg)](https://pandas.pydata.org)
 [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/pydata/pandas)
 [![Powered by NumFOCUS](https://img.shields.io/badge/powered%20by-NumFOCUS-orange.svg?style=flat&colorA=E1523D&colorB=007D8A)](https://numfocus.org)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
 ## What is it?
 

doc/source/ecosystem.rst

@@ -30,7 +30,7 @@ substantial projects that you feel should be on this list, please let us know.
 Data cleaning and validation
 ----------------------------
 
-`pyjanitor <https://github.com/ericmjl/pyjanitor/>`__
+`Pyjanitor <https://github.com/ericmjl/pyjanitor/>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Pyjanitor provides a clean API for cleaning data, using method chaining.
@@ -115,7 +115,7 @@ It is very similar to the matplotlib plotting backend, but provides interactive
 web-based charts and maps.
 
 
-`seaborn <https://seaborn.pydata.org>`__
+`Seaborn <https://seaborn.pydata.org>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Seaborn is a Python visualization library based on
@@ -136,7 +136,7 @@ provides a powerful, declarative and extremely general way to generate bespoke p
 Various implementations to other languages are available.
 A good implementation for Python users is `has2k1/plotnine <https://github.com/has2k1/plotnine/>`__.
 
-`IPython Vega <https://github.com/vega/ipyvega>`__
+`IPython vega <https://github.com/vega/ipyvega>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 `IPython Vega <https://github.com/vega/ipyvega>`__ leverages `Vega
@@ -147,7 +147,7 @@ A good implementation for Python users is `has2k1/plotnine <https://github.com/h
 
 `Plotly’s <https://plot.ly/>`__ `Python API <https://plot.ly/python/>`__ enables interactive figures and web shareability. Maps, 2D, 3D, and live-streaming graphs are rendered with WebGL and `D3.js <https://d3js.org/>`__. The library supports plotting directly from a pandas DataFrame and cloud-based collaboration. Users of `matplotlib, ggplot for Python, and Seaborn <https://plot.ly/python/matplotlib-to-plotly-tutorial/>`__ can convert figures into interactive web-based plots. Plots can be drawn in `IPython Notebooks <https://plot.ly/ipython-notebooks/>`__ , edited with R or MATLAB, modified in a GUI, or embedded in apps and dashboards. Plotly is free for unlimited sharing, and has `cloud <https://plot.ly/product/plans/>`__, `offline <https://plot.ly/python/offline/>`__, or `on-premise <https://plot.ly/product/enterprise/>`__ accounts for private use.
 
-`QtPandas <https://github.com/draperjames/qtpandas>`__
+`Qtpandas <https://github.com/draperjames/qtpandas>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Spun off from the main pandas library, the `qtpandas <https://github.com/draperjames/qtpandas>`__
@@ -187,7 +187,7 @@ See :ref:`Options and Settings <options>` and
 :ref:`Available Options <options.available>`
 for pandas ``display.`` settings.
 
-`quantopian/qgrid <https://github.com/quantopian/qgrid>`__
+`Quantopian/qgrid <https://github.com/quantopian/qgrid>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 qgrid is "an interactive grid for sorting and filtering
@@ -249,12 +249,12 @@ The following data feeds are available:
  * Stooq Index Data
  * MOEX Data
 
-`quandl/Python <https://github.com/quandl/Python>`__
+`Quandl/Python <https://github.com/quandl/Python>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Quandl API for Python wraps the Quandl REST API to return
 Pandas DataFrames with timeseries indexes.
 
-`pydatastream <https://github.com/vfilimonov/pydatastream>`__
+`Pydatastream <https://github.com/vfilimonov/pydatastream>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 PyDatastream is a Python interface to the
 `Refinitiv Datastream (DWS) <https://www.refinitiv.com/en/products/datastream-macroeconomic-analysis>`__
@@ -384,7 +384,7 @@ Pandas provides an interface for defining
 system. The following libraries implement that interface to provide types not
 found in NumPy or pandas, which work well with pandas' data containers.
 
-`cyberpandas`_
+`Cyberpandas`_
 ~~~~~~~~~~~~~~
 
 Cyberpandas provides an extension type for storing arrays of IP Addresses. These
@@ -411,4 +411,4 @@ Library         Accessor   Classes                   Description
 .. _pdvega: https://altair-viz.github.io/pdvega/
 .. _Altair: https://altair-viz.github.io/
 .. _pandas_path: https://github.com/drivendataorg/pandas-path/
-.. _pathlib.Path: https://docs.python.org/3/library/pathlib.html
+.. _pathlib.Path: https://docs.python.org/3/library/pathlib.html

doc/source/reference/offset_frequency.rst

@@ -1044,6 +1044,7 @@ Properties
     Tick.nanos
     Tick.normalize
     Tick.rule_code
+    Tick.n
 
 Methods
 ~~~~~~~
@@ -1077,6 +1078,7 @@ Properties
     Day.nanos
     Day.normalize
     Day.rule_code
+    Day.n
 
 Methods
 ~~~~~~~
@@ -1110,6 +1112,7 @@ Properties
     Hour.nanos
     Hour.normalize
     Hour.rule_code
+    Hour.n
 
 Methods
 ~~~~~~~
@@ -1143,6 +1146,7 @@ Properties
     Minute.nanos
     Minute.normalize
     Minute.rule_code
+    Minute.n
 
 Methods
 ~~~~~~~
@@ -1176,6 +1180,7 @@ Properties
     Second.nanos
     Second.normalize
     Second.rule_code
+    Second.n
 
 Methods
 ~~~~~~~
@@ -1209,6 +1214,7 @@ Properties
     Milli.nanos
     Milli.normalize
     Milli.rule_code
+    Milli.n
 
 Methods
 ~~~~~~~
@@ -1242,6 +1248,7 @@ Properties
     Micro.nanos
     Micro.normalize
     Micro.rule_code
+    Micro.n
 
 Methods
 ~~~~~~~
@@ -1275,6 +1282,7 @@ Properties
     Nano.nanos
     Nano.normalize
     Nano.rule_code
+    Nano.n
 
 Methods
 ~~~~~~~

doc/source/user_guide/computation.rst

@@ -648,6 +648,24 @@ from present information back to past information. This allows the rolling windo
 Currently, this feature is only implemented for time-based windows.
 For fixed windows, the closed parameter cannot be set and the rolling window will always have both endpoints closed.
 
+.. _stats.iter_rolling_window:
+
+Iteration over window:
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 1.1.0
+
+``Rolling`` and ``Expanding`` objects now support iteration. Be noted that ``min_periods`` is ignored in iteration.
+
+.. ipython::
+
+   In [1]: df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
+
+   In [2]: for i in df.rolling(2):
+      ...:     print(i)
+      ...:
+
+
 .. _stats.moments.ts-versus-resampling:
 
 Time-aware rolling vs. resampling

doc/source/whatsnew/v1.1.0.rst

@@ -235,6 +235,7 @@ Other enhancements
   :class:`~pandas.io.stata.StataWriter`, :class:`~pandas.io.stata.StataWriter117`,
   and :class:`~pandas.io.stata.StataWriterUTF8` (:issue:`26599`).
 - :meth:`HDFStore.put` now accepts `track_times` parameter. Parameter is passed to ``create_table`` method of ``PyTables`` (:issue:`32682`).
+- Make :class:`pandas.core.window.Rolling` and :class:`pandas.core.window.Expanding` iterable（:issue:`11704`)
 
 .. ---------------------------------------------------------------------------
 
@@ -585,6 +586,7 @@ Deprecations
 
 - :func:`pandas.api.types.is_categorical` is deprecated and will be removed in a future version; use `:func:pandas.api.types.is_categorical_dtype` instead (:issue:`33385`)
 - :meth:`Index.get_value` is deprecated and will be removed in a future version (:issue:`19728`)
+- :meth:`DateOffset.__call__` is deprecated and will be removed in a future version, use ``offset + other`` instead (:issue:`34171`)
 
 .. ---------------------------------------------------------------------------
 
@@ -605,6 +607,8 @@ Performance improvements
   sparse values from ``scipy.sparse`` matrices using the
   :meth:`DataFrame.sparse.from_spmatrix` constructor (:issue:`32821`,
   :issue:`32825`,  :issue:`32826`, :issue:`32856`, :issue:`32858`).
+- Performance improvement for groupby methods :meth:`~pandas.core.groupby.groupby.Groupby.first`
+  and :meth:`~pandas.core.groupby.groupby.Groupby.last` (:issue:`34178`)
 - Performance improvement in :func:`factorize` for nullable (integer and boolean) dtypes (:issue:`33064`).
 - Performance improvement in reductions (sum, prod, min, max) for nullable (integer and boolean) dtypes (:issue:`30982`, :issue:`33261`, :issue:`33442`).
 
@@ -813,6 +817,8 @@ Groupby/resample/rolling
 - Bug in :meth:`GroupBy.first` and :meth:`GroupBy.last` where None is not preserved in object dtype (:issue:`32800`)
 - Bug in :meth:`Rolling.min` and :meth:`Rolling.max`: Growing memory usage after multiple calls when using a fixed window (:issue:`30726`)
 - Bug in :meth:`GroupBy.agg`, :meth:`GroupBy.transform`, and :meth:`GroupBy.resample` where subclasses are not preserved (:issue:`28330`)
+- Bug in :meth:`GroupBy.rolling.apply` ignores args and kwargs parameters (:issue:`33433`)
+
 
 Reshaping
 ^^^^^^^^^

pandas/_libs/groupby.pyx

@@ -9,11 +9,9 @@ cimport numpy as cnp
 from numpy cimport (ndarray,
                     int8_t, int16_t, int32_t, int64_t, uint8_t, uint16_t,
                     uint32_t, uint64_t, float32_t, float64_t, complex64_t, complex128_t)
+from numpy.math cimport NAN
 cnp.import_array()
 
-cdef extern from "numpy/npy_math.h":
-    float64_t NAN "NPY_NAN"
-
 from pandas._libs.util cimport numeric, get_nat
 
 from pandas._libs.algos cimport (swap, TiebreakEnumType, TIEBREAK_AVERAGE,

pandas/_libs/hashtable.pyx

@@ -8,10 +8,9 @@ from libc.stdlib cimport malloc, free
 import numpy as np
 cimport numpy as cnp
 from numpy cimport ndarray, uint8_t, uint32_t, float64_t
+from numpy.math cimport NAN
 cnp.import_array()
 
-cdef extern from "numpy/npy_math.h":
-    float64_t NAN "NPY_NAN"
 
 from pandas._libs.khash cimport (
     khiter_t,
@@ -54,7 +53,7 @@ from pandas._libs.khash cimport (
 )
 
 
-cimport pandas._libs.util as util
+from pandas._libs cimport util
 
 from pandas._libs.missing cimport checknull
 

pandas/_libs/index.pyx

@@ -19,11 +19,10 @@ from numpy cimport (
 cnp.import_array()
 
 
-cimport pandas._libs.util as util
+from pandas._libs cimport util
 
-from pandas._libs.tslibs import Period, Timedelta
 from pandas._libs.tslibs.nattype cimport c_NaT as NaT
-from pandas._libs.tslibs.base cimport ABCTimestamp
+from pandas._libs.tslibs.base cimport ABCTimestamp, ABCTimedelta, ABCPeriod
 
 from pandas._libs.hashtable cimport HashTable
 
@@ -470,7 +469,7 @@ cdef class TimedeltaEngine(DatetimeEngine):
         return 'm8[ns]'
 
     cdef int64_t _unbox_scalar(self, scalar) except? -1:
-        if not (isinstance(scalar, Timedelta) or scalar is NaT):
+        if not (isinstance(scalar, ABCTimedelta) or scalar is NaT):
             raise TypeError(scalar)
         return scalar.value
 
@@ -480,7 +479,7 @@ cdef class PeriodEngine(Int64Engine):
     cdef int64_t _unbox_scalar(self, scalar) except? -1:
         if scalar is NaT:
             return scalar.value
-        if isinstance(scalar, Period):
+        if isinstance(scalar, ABCPeriod):
             # NB: we assume that we have the correct freq here.
             return scalar.ordinal
         raise TypeError(scalar)

pandas/_libs/internals.pyx

@@ -1,5 +1,6 @@
-import cython
 from collections import defaultdict
+
+import cython
 from cython import Py_ssize_t
 
 from cpython.slice cimport PySlice_GetIndicesEx

pandas/_libs/interval.pyx

@@ -33,7 +33,7 @@ from numpy cimport (
 cnp.import_array()
 
 
-cimport pandas._libs.util as util
+from pandas._libs cimport util
 
 from pandas._libs.hashtable cimport Int64Vector
 from pandas._libs.tslibs.util cimport (
@@ -42,8 +42,7 @@ from pandas._libs.tslibs.util cimport (
     is_timedelta64_object,
 )
 
-from pandas._libs.tslibs import Timestamp
-from pandas._libs.tslibs.timedeltas import Timedelta
+from pandas._libs.tslibs.base cimport ABCTimestamp, ABCTimedelta
 from pandas._libs.tslibs.timezones cimport tz_compare
 
 
@@ -329,7 +328,7 @@ cdef class Interval(IntervalMixin):
             raise ValueError(f"invalid option for 'closed': {closed}")
         if not left <= right:
             raise ValueError("left side of interval must be <= right side")
-        if (isinstance(left, Timestamp) and
+        if (isinstance(left, ABCTimestamp) and
                 not tz_compare(left.tzinfo, right.tzinfo)):
             # GH 18538
             raise ValueError("left and right must have the same time zone, got "
@@ -341,7 +340,7 @@ cdef class Interval(IntervalMixin):
     def _validate_endpoint(self, endpoint):
         # GH 23013
         if not (is_integer_object(endpoint) or is_float_object(endpoint) or
-                isinstance(endpoint, (Timestamp, Timedelta))):
+                isinstance(endpoint, (ABCTimestamp, ABCTimedelta))):
             raise ValueError("Only numeric, Timestamp and Timedelta endpoints "
                              "are allowed when constructing an Interval.")
 
@@ -371,7 +370,7 @@ cdef class Interval(IntervalMixin):
         right = self.right
 
         # TODO: need more general formatting methodology here
-        if isinstance(left, Timestamp) and isinstance(right, Timestamp):
+        if isinstance(left, ABCTimestamp) and isinstance(right, ABCTimestamp):
             left = left._short_repr
             right = right._short_repr
 

pandas/_libs/lib.pyx

@@ -1,6 +1,5 @@
 from collections import abc
 from decimal import Decimal
-
 import warnings
 
 import cython
@@ -63,7 +62,7 @@ cdef extern from "numpy/arrayobject.h":
 cdef extern from "src/parse_helper.h":
     int floatify(object, float64_t *result, int *maybe_int) except -1
 
-cimport pandas._libs.util as util
+from pandas._libs cimport util
 from pandas._libs.util cimport is_nan, UINT64_MAX, INT64_MAX, INT64_MIN
 
 from pandas._libs.tslib import array_to_datetime

pandas/_libs/missing.pyx

@@ -8,7 +8,7 @@ cimport numpy as cnp
 from numpy cimport ndarray, int64_t, uint8_t, float64_t
 cnp.import_array()
 
-cimport pandas._libs.util as util
+from pandas._libs cimport util
 
 
 from pandas._libs.tslibs.np_datetime cimport get_datetime64_value, get_timedelta64_value

pandas/_libs/parsers.pyx

@@ -34,7 +34,7 @@ cimport numpy as cnp
 from numpy cimport ndarray, uint8_t, uint64_t, int64_t, float64_t
 cnp.import_array()
 
-cimport pandas._libs.util as util
+from pandas._libs cimport util
 from pandas._libs.util cimport UINT64_MAX, INT64_MAX, INT64_MIN
 import pandas._libs.lib as lib
 

pandas/_libs/reduction.pyx

@@ -14,7 +14,7 @@ from numpy cimport (ndarray,
                     flatiter)
 cnp.import_array()
 
-cimport pandas._libs.util as util
+from pandas._libs cimport util
 from pandas._libs.lib import maybe_convert_objects, is_scalar
 
 
@@ -603,7 +603,7 @@ cdef class BlockSlider:
             arr.shape[1] = 0
 
 
-def compute_reduction(arr: np.ndarray, f, axis: int = 0, dummy=None, labels=None):
+def compute_reduction(arr: ndarray, f, axis: int = 0, dummy=None, labels=None):
     """
 
     Parameters