Merge remote-tracking branch 'upstream/master' into integer-array-pow-2

Author: TomAugspurger

ci/deps/azure-36-locale_slow.yaml

@@ -18,7 +18,7 @@ dependencies:
   - lxml
   - matplotlib=2.2.2
   - numpy=1.14.*
-  - openpyxl=2.4.8
+  - openpyxl=2.5.7
   - python-dateutil
   - python-blosc
   - pytz=2017.2

ci/deps/azure-36-minimum_versions.yaml

@@ -11,14 +11,15 @@ dependencies:
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0
   - pytest-azurepipelines
+  - psutil
 
   # pandas dependencies
   - beautifulsoup4=4.6.0
   - bottleneck=1.2.1
   - jinja2=2.8
   - numexpr=2.6.2
   - numpy=1.13.3
-  - openpyxl=2.4.8
+  - openpyxl=2.5.7
   - pytables=3.4.2
   - python-dateutil=2.6.1
   - pytz=2017.2

doc/source/getting_started/install.rst

@@ -255,10 +255,10 @@ gcsfs                     0.2.2              Google Cloud Storage access
 html5lib                                     HTML parser for read_html (see :ref:`note <optional_html>`)
 lxml                      3.8.0              HTML parser for read_html (see :ref:`note <optional_html>`)
 matplotlib                2.2.2              Visualization
-openpyxl                  2.4.8              Reading / writing for xlsx files
+openpyxl                  2.5.7              Reading / writing for xlsx files
 pandas-gbq                0.8.0              Google Big Query access
 psycopg2                                     PostgreSQL engine for sqlalchemy
-pyarrow                   0.12.0             Parquet and feather reading / writing
+pyarrow                   0.12.0             Parquet, ORC (requires 0.13.0), and feather reading / writing
 pymysql                   0.7.11             MySQL engine for sqlalchemy
 pyreadstat                                   SPSS files (.sav) reading
 pytables                  3.4.2              HDF5 reading / writing

doc/source/reference/io.rst

@@ -98,6 +98,13 @@ Parquet
 
    read_parquet
 
+ORC
+~~~
+.. autosummary::
+   :toctree: api/
+
+   read_orc
+
 SAS
 ~~~
 .. autosummary::

doc/source/user_guide/io.rst

@@ -28,6 +28,7 @@ The pandas I/O API is a set of top level ``reader`` functions accessed like
     binary;`HDF5 Format <https://support.hdfgroup.org/HDF5/whatishdf5.html>`__;:ref:`read_hdf<io.hdf5>`;:ref:`to_hdf<io.hdf5>`
     binary;`Feather Format <https://github.com/wesm/feather>`__;:ref:`read_feather<io.feather>`;:ref:`to_feather<io.feather>`
     binary;`Parquet Format <https://parquet.apache.org/>`__;:ref:`read_parquet<io.parquet>`;:ref:`to_parquet<io.parquet>`
+    binary;`ORC Format <//https://orc.apache.org/>`__;:ref:`read_orc<io.orc>`;
     binary;`Msgpack <https://msgpack.org/index.html>`__;:ref:`read_msgpack<io.msgpack>`;:ref:`to_msgpack<io.msgpack>`
     binary;`Stata <https://en.wikipedia.org/wiki/Stata>`__;:ref:`read_stata<io.stata_reader>`;:ref:`to_stata<io.stata_writer>`
     binary;`SAS <https://en.wikipedia.org/wiki/SAS_(software)>`__;:ref:`read_sas<io.sas_reader>`;
@@ -4858,6 +4859,17 @@ The above example creates a partitioned dataset that may look like:
    except OSError:
        pass
 
+.. _io.orc:
+
+ORC
+---
+
+.. versionadded:: 1.0.0
+
+Similar to the :ref:`parquet <io.parquet>` format, the `ORC Format <//https://orc.apache.org/>`__ is a binary columnar serialization
+for data frames. It is designed to make reading data frames efficient. Pandas provides *only* a reader for the
+ORC format, :func:`~pandas.read_orc`. This requires the `pyarrow <https://arrow.apache.org/docs/python/>`__ library.
+
 .. _io.sql:
 
 SQL queries
@@ -5761,6 +5773,3 @@ Space on disk (in bytes)
     24009288 Oct 10 06:43 test_fixed_compress.hdf
     24458940 Oct 10 06:44 test_table.hdf
     24458940 Oct 10 06:44 test_table_compress.hdf
-
-
-

doc/source/whatsnew/v1.0.0.rst

@@ -424,7 +424,7 @@ Optional libraries below the lowest tested version may still work, but are not c
 +-----------------+-----------------+---------+
 | matplotlib      | 2.2.2           |         |
 +-----------------+-----------------+---------+
-| openpyxl        | 2.4.8           |         |
+| openpyxl        | 2.5.7           |    X    |
 +-----------------+-----------------+---------+
 | pyarrow         | 0.12.0          |    X    |
 +-----------------+-----------------+---------+
@@ -541,7 +541,7 @@ or ``matplotlib.Axes.plot``. See :ref:`plotting.formatters` for more.
 - Removed the previously deprecated :meth:`Series.compound` and :meth:`DataFrame.compound` (:issue:`26405`)
 - Changed the the default value of `inplace` in :meth:`DataFrame.set_index` and :meth:`Series.set_axis`. It now defaults to ``False`` (:issue:`27600`)
 - Removed the previously deprecated :attr:`Series.cat.categorical`, :attr:`Series.cat.index`, :attr:`Series.cat.name` (:issue:`24751`)
-- :func:`to_datetime` no longer accepts "box" argument, always returns :class:`DatetimeIndex` or :class:`Index`, :class:`Series`, or :class:`DataFrame` (:issue:`24486`)
+- :func:`to_datetime` and :func:`to_timedelta` no longer accept "box" argument, always returns :class:`DatetimeIndex`, :class:`TimedeltaIndex`, :class:`Index`, :class:`Series`, or :class:`DataFrame` (:issue:`24486`)
 - :func:`to_timedelta`, :class:`Timedelta`, and :class:`TimedeltaIndex` no longer allow "M", "y", or "Y" for the "unit" argument (:issue:`23264`)
 - Removed the previously deprecated ``time_rule`` keyword from (non-public) :func:`offsets.generate_range`, which has been moved to :func:`core.arrays._ranges.generate_range` (:issue:`24157`)
 - :meth:`DataFrame.loc` or :meth:`Series.loc` with listlike indexers and missing labels will no longer reindex (:issue:`17295`)
@@ -713,6 +713,7 @@ Numeric
 - Bug in :class:`NumericIndex` construction that caused indexing to fail when integers in the ``np.uint64`` range were used (:issue:`28023`)
 - Bug in :class:`NumericIndex` construction that caused :class:`UInt64Index` to be casted to :class:`Float64Index` when integers in the ``np.uint64`` range were used to index a :class:`DataFrame` (:issue:`28279`)
 - Bug in :meth:`Series.interpolate` when using method=`index` with an unsorted index, would previously return incorrect results. (:issue:`21037`)
+- Bug in :meth:`DataFrame.round` where a :class:`DataFrame` with a :class:`CategoricalIndex` of :class:`IntervalIndex` columns would incorrectly raise a ``TypeError`` (:issue:`30063`)
 
 Conversion
 ^^^^^^^^^^
@@ -730,7 +731,7 @@ Strings
 Interval
 ^^^^^^^^
 
--
+- Bug in :meth:`IntervalIndex.get_indexer` where a :class:`Categorical` or :class:`CategoricalIndex` ``target`` would incorrectly raise a ``TypeError`` (:issue:`30063`)
 -
 
 Indexing
@@ -742,6 +743,7 @@ Indexing
 - Fix assignment of column via `.loc` with numpy non-ns datetime type (:issue:`27395`)
 - Bug in :meth:`Float64Index.astype` where ``np.inf`` was not handled properly when casting to an integer dtype (:issue:`28475`)
 - :meth:`Index.union` could fail when the left contained duplicates (:issue:`28257`)
+- Bug when indexing with ``.loc`` where the index was a :class:`CategoricalIndex` with integer and float categories, a ValueError was raised (:issue:`17569`)
 - :meth:`Index.get_indexer_non_unique` could fail with `TypeError` in some cases, such as when searching for ints in a string index (:issue:`28257`)
 - Bug in :meth:`Float64Index.get_loc` incorrectly raising ``TypeError`` instead of ``KeyError`` (:issue:`29189`)
 
@@ -858,7 +860,7 @@ Other
 - Bug in :meth:`DataFrame.append` that raised ``IndexError`` when appending with empty list (:issue:`28769`)
 - Fix :class:`AbstractHolidayCalendar` to return correct results for
   years after 2030 (now goes up to 2200) (:issue:`27790`)
-- Fixed :class:`IntegerArray` returning ``NA`` rather than ``inf`` for operations dividing by 0 (:issue:`27398`)
+- Fixed :class:`IntegerArray` returning ``inf`` rather than ``NA`` for operations dividing by 0 (:issue:`27398`)
 - Fixed ``pow`` operations for :class:`IntegerArray` when the other value is ``0`` or ``1`` (:issue:`29997`)
 - Bug in :meth:`Series.count` raises if use_inf_as_na is enabled (:issue:`29478`)
 

pandas/__init__.py

@@ -168,6 +168,7 @@
     # misc
     read_clipboard,
     read_parquet,
+    read_orc,
     read_feather,
     read_gbq,
     read_html,

pandas/_libs/testing.pyx

@@ -66,7 +66,8 @@ cpdef assert_almost_equal(a, b,
                           check_less_precise=False,
                           bint check_dtype=True,
                           obj=None, lobj=None, robj=None):
-    """Check that left and right objects are almost equal.
+    """
+    Check that left and right objects are almost equal.
 
     Parameters
     ----------
@@ -89,7 +90,6 @@ cpdef assert_almost_equal(a, b,
         Specify right object name being compared, internally used to show
         appropriate assertion message
     """
-
     cdef:
         int decimal
         double diff = 0.0
@@ -127,9 +127,9 @@ cpdef assert_almost_equal(a, b,
             # classes can't be the same, to raise error
             assert_class_equal(a, b, obj=obj)
 
-        assert has_length(a) and has_length(b), (
-            f"Can't compare objects without length, one or both is invalid: "
-            f"({a}, {b})")
+        assert has_length(a) and has_length(b), ("Can't compare objects without "
+                                                 "length, one or both is invalid: "
+                                                 f"({a}, {b})")
 
         if a_is_ndarray and b_is_ndarray:
             na, nb = a.size, b.size
@@ -157,7 +157,7 @@ cpdef assert_almost_equal(a, b,
             else:
                 r = None
 
-            raise_assert_detail(obj, f'{obj} length are different', na, nb, r)
+            raise_assert_detail(obj, f"{obj} length are different", na, nb, r)
 
         for i in xrange(len(a)):
             try:
@@ -169,8 +169,8 @@ cpdef assert_almost_equal(a, b,
 
         if is_unequal:
             from pandas.util.testing import raise_assert_detail
-            msg = (f'{obj} values are different '
-                   f'({np.round(diff * 100.0 / na, 5)} %)')
+            msg = (f"{obj} values are different "
+                   f"({np.round(diff * 100.0 / na, 5)} %)")
             raise_assert_detail(obj, msg, lobj, robj)
 
         return True

pandas/_libs/tslibs/timedeltas.pyx

@@ -1460,7 +1460,7 @@ class Timedelta(_Timedelta):
                 # also timedelta-like
                 return _broadcast_floordiv_td64(self.value, other, _rfloordiv)
 
-            # Includes integer array // Timedelta, deprecated in GH#19761
+            # Includes integer array // Timedelta, disallowed in GH#19761
             raise TypeError(f'Invalid dtype {other.dtype} for __floordiv__')
 
         elif is_float_object(other) and util.is_nan(other):

pandas/compat/_optional.py

@@ -14,7 +14,7 @@
     "matplotlib": "2.2.2",
     "numexpr": "2.6.2",
     "odfpy": "1.3.0",
-    "openpyxl": "2.4.8",
+    "openpyxl": "2.5.7",
     "pandas_gbq": "0.8.0",
     "pyarrow": "0.12.0",
     "pytables": "3.4.2",

pandas/compat/pickle_compat.py

@@ -4,7 +4,7 @@
 
 import copy
 import pickle as pkl
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Optional
 import warnings
 
 from pandas import Index
@@ -219,8 +219,9 @@ def load_newobj_ex(self):
     pass
 
 
-def load(fh, encoding=None, is_verbose=False):
-    """load a pickle, with a provided encoding
+def load(fh, encoding: Optional[str] = None, is_verbose: bool = False):
+    """
+    Load a pickle, with a provided encoding,
 
     Parameters
     ----------