Skip to content

Commit 3891ff1

Browse files
authored
Merge pull request #634 from AndreiBarsan/doc_pyarrow_compat_warning
Update documentation to better reflect how PyArrow interoperability works
2 parents f9ed786 + a0e2aa5 commit 3891ff1

File tree

1 file changed

+14
-7
lines changed

1 file changed

+14
-7
lines changed

docs/source/features.rst

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -94,13 +94,20 @@ object.
9494
PyArrow integration
9595
-------------------
9696

97-
`pyarrow`_ has its own internal idea of what a file-system is (``pyarrow.filesystem.FileSystem``),
97+
`pyarrow`_ has its own internal idea of what a file-system is (``pyarrow.fs.FileSystem``),
9898
and some functions, particularly the loading of parquet, require that the target be compatible.
99-
As it happens, the design of the file-system interface in ``pyarrow`` *is* compatible with `fsspec`
100-
(this is not by accident). Therefore at import time, ``fsspec`` checks for the existence of
101-
``pyarrow``, and, if found, adds it to the superclasses of the spec base-class. In this manner,
102-
all ``fsspec``-derived file-systems are also pyarrow file-systems, and can be used by pyarrow
103-
functions.
99+
As it happens, the design of the file-system interface in ``pyarrow`` *is* compatible with ``fsspec``
100+
(this is not by accident).
101+
102+
At import time, ``fsspec`` checks for the existence of ``pyarrow``, and, if ``pyarrow < 2.0`` is
103+
found, adds its base filesystem to the superclasses of the spec base-class.
104+
For ``pyarrow >= 2.0``, ``fsspec`` file systems can simply be passed to ``pyarrow`` functions
105+
that expect ``pyarrow`` filesystems, and ``pyarrow`` `will automatically wrap them
106+
<https://arrow.apache.org/docs/python/filesystems.html#using-fsspec-compatible-filesystems>`_.
107+
108+
In this manner, all ``fsspec``-derived file-systems are also ``pyarrow`` file-systems, and can be used
109+
by ``pyarrow`` functions.
110+
104111

105112
.. _pyarrow: https://arrow.apache.org/docs/python/
106113

@@ -377,7 +384,7 @@ Obviously, you should only define default values that are appropriate for
377384
a given file system implementation. INI files only support string values.
378385

379386
Alternatively, you can provide overrides with environment variables of
380-
the style "FSSPEC_{protocol}_{kwargname}=value".
387+
the style ``FSSPEC_{protocol}_{kwargname}=value``.
381388

382389
Configuration is determined in the following order, with later items winning:
383390

0 commit comments

Comments
 (0)