Skip to content

DOC: Add expanded index descriptors for specifying for RangeIndex-as-metadata in Parquet file schema #25709

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Aug 8, 2019
Merged

DOC: Add expanded index descriptors for specifying for RangeIndex-as-metadata in Parquet file schema #25709

Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
766aa50
Add specification for RangeIndex-as-metadata in Parquet file schema c…
wesm Mar 13, 2019
2c8431c
Add string quotes to range
wesm Mar 13, 2019
931ca2c
Update doc/source/development/developer.rst
TomAugspurger Mar 13, 2019
d3cd904
Revert to current scheme of index column names in 'index_columns' for…
wesm Apr 2, 2019
8f6d6a7
Merge remote-tracking branch 'upstream/master' into GH-25672-parquet-…
jorisvandenbossche Aug 8, 2019
10d1e86
small clean-up
jorisvandenbossche Aug 8, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 47 additions & 19 deletions doc/source/development/developer.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,17 @@ So that a ``pandas.DataFrame`` can be faithfully reconstructed, we store a

.. code-block:: text

{'index_columns': ['__index_level_0__', '__index_level_1__', ...],
{'index_columns': [<descr0>, <descr1>, ...],
'column_indexes': [<ci0>, <ci1>, ..., <ciN>],
'columns': [<c0>, <c1>, ...],
'pandas_version': $VERSION}
'pandas_version': $VERSION,
'creator': {
'library': $LIBRARY,
'version': $LIBRARY_VERSION
}}

The "descriptor" values ``<descr0>`` in the ``'index_columns'`` field are
dictionaries with values as described below.

Here, ``<c0>``/``<ci0>`` and so forth are dictionaries containing the metadata
for each column, *including the index columns*. This has JSON form:
Expand All @@ -53,26 +60,42 @@ for each column, *including the index columns*. This has JSON form:
'numpy_type': numpy_type,
'metadata': metadata}

.. note::
See below for the detailed specification for these

Index Metadata Descriptors
~~~~~~~~~~~~~~~~~~~~~~~~~~

``RangeIndex`` can be stored as metadata only, not requiring serialization. The
descriptor format for these as is follows:

.. code-block:: python

{'kind': 'range',
'name': index.name,
'start': index._start,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#25720 was just merged, prob could just change this (though of course implementing this for <0.25.x will still require this)

'stop': index._stop,
'step': index._step}

Every index column is stored with a name matching the pattern
``__index_level_\d+__`` and its corresponding column information is can be
found with the following code snippet.
Other index types must be serialized as data columns along with the other
DataFrame columns. The metadata for these is a dict with ``kind`` field
``'serialized'`` and ``'field_name'`` field indicating which data column
contains the index data. For example,

Following this naming convention isn't strictly necessary, but strongly
suggested for compatibility with Arrow.
.. code-block:: python

Here's an example of how the index metadata is structured in pyarrow:
{'kind': 'serialized',
'field_name': '__index_level_0__'}

.. code-block:: python
Every index column is stored with a name matching the pattern
``__index_level_\d+__``. Following this naming convention isn't strictly
necessary, but strongly suggested for compatibility with Arrow and
disambiguation. The ``'field_name'`` is the actual name of the column in the
serialized Parquet table. If the ``Index`` has a non-None ``name`` attribute,
then it can be found in the ``name`` field of the metadata for that serialized
data column as described below.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this actually correct? (although it was already there in the doc before, to be clear)
This seems to indicate that an index always gets a __index_level_x__ name as the field_name, regardless of the name it has (so not only if it is None).

But this is not what I see from a quick test:

In [3]: pyarrow.__version__                                                                                                                                                     
Out[3]: '0.12.0'

In [4]: df = pd.DataFrame({'a': [1, 2, 3], 'b': [3, 4, 5]}).set_index('a')                                                                                                      

In [5]: df                                                                                                                                                                      
Out[5]: 
   b
a   
1  3
2  4
3  5

In [6]: pyarrow.Table.from_pandas(df)                                                                                                                                           
Out[6]: 
pyarrow.Table
b: int64
a: int64
metadata
--------
OrderedDict([(b'pandas',
              b'{"index_columns": ["a"], "column_indexes": [{"name": null, "'
              b'field_name": null, "pandas_type": "unicode", "numpy_type": "'
              b'object", "metadata": {"encoding": "UTF-8"}}], "columns": [{"'
              b'name": "b", "field_name": "b", "pandas_type": "int64", "nump'
              b'y_type": "int64", "metadata": null}, {"name": "a", "field_na'
              b'me": "a", "pandas_type": "int64", "numpy_type": "int64", "me'
              b'tadata": null}], "pandas_version": "0.23.4"}')])

(I remember that we had a discussion about this before, but can't directly remember the outcome of that)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears that the current behavior is to use the name of the index if it does not conflict with any of the data columns. So we should update these docs to reflect that


# assuming there's at least 3 levels in the index
index_columns = metadata['index_columns'] # noqa: F821
columns = metadata['columns'] # noqa: F821
ith_index = 2
assert index_columns[ith_index] == '__index_level_2__'
ith_index_info = columns[-len(index_columns):][ith_index]
ith_index_level_name = ith_index_info['name']
Column Metadata
~~~~~~~~~~~~~~~

``pandas_type`` is the logical type of the column, and is one of:

Expand Down Expand Up @@ -121,7 +144,8 @@ As an example of fully-formed metadata:

.. code-block:: text

{'index_columns': ['__index_level_0__'],
{'index_columns': [{'kind': 'serialized',
'field_name': '__index_level_0__'}],
'column_indexes': [
{'name': None,
'field_name': 'None',
Expand Down Expand Up @@ -161,4 +185,8 @@ As an example of fully-formed metadata:
'numpy_type': 'int64',
'metadata': None}
],
'pandas_version': '0.20.0'}
'pandas_version': '0.20.0',
'creator': {
'library': 'pyarrow',
'version': '0.13.0'
}}