From 2512ac890a4fe046e102d4538bddd2d203478511 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andrei=20B=C3=A2rsan?= Date: Wed, 12 May 2021 18:16:39 -0400 Subject: [PATCH 1/3] Mention incompat with pyarrow >= 2.0; minor formatting fixes --- docs/source/features.rst | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/source/features.rst b/docs/source/features.rst index 473548950..44c3f0b0a 100644 --- a/docs/source/features.rst +++ b/docs/source/features.rst @@ -94,9 +94,16 @@ object. PyArrow integration ------------------- +.. warning:: + ``fsspec`` is only compatible with PyArrow's classic ``FileSystem``. Starting with version + 2.0 (released in October 2020), `pyarrow`_ switched to a C++ filesystem implementation that + is no longer compatible with ``fsspec``. Please refer to + `this GitHub issue https://github.com/intake/filesystem_spec/issues/295`_ for additional details. + + `pyarrow`_ has its own internal idea of what a file-system is (``pyarrow.filesystem.FileSystem``), and some functions, particularly the loading of parquet, require that the target be compatible. -As it happens, the design of the file-system interface in ``pyarrow`` *is* compatible with `fsspec` +As it happens, the design of the file-system interface in ``pyarrow`` *is* compatible with ``fsspec`` (this is not by accident). Therefore at import time, ``fsspec`` checks for the existence of ``pyarrow``, and, if found, adds it to the superclasses of the spec base-class. In this manner, all ``fsspec``-derived file-systems are also pyarrow file-systems, and can be used by pyarrow @@ -377,7 +384,7 @@ Obviously, you should only define default values that are appropriate for a given file system implementation. INI files only support string values. Alternatively, you can provide overrides with environment variables of -the style "FSSPEC_{protocol}_{kwargname}=value". +the style ``FSSPEC_{protocol}_{kwargname}=value``. Configuration is determined in the following order, with later items winning: From aabb418be044045c3b38f818c8fbf47beedeab2e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andrei=20B=C3=A2rsan?= Date: Wed, 12 May 2021 18:22:24 -0400 Subject: [PATCH 2/3] Fix rst link syntax --- docs/source/features.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/features.rst b/docs/source/features.rst index 44c3f0b0a..2138d3810 100644 --- a/docs/source/features.rst +++ b/docs/source/features.rst @@ -98,7 +98,7 @@ PyArrow integration ``fsspec`` is only compatible with PyArrow's classic ``FileSystem``. Starting with version 2.0 (released in October 2020), `pyarrow`_ switched to a C++ filesystem implementation that is no longer compatible with ``fsspec``. Please refer to - `this GitHub issue https://github.com/intake/filesystem_spec/issues/295`_ for additional details. + `this GitHub issue `_ for additional details. `pyarrow`_ has its own internal idea of what a file-system is (``pyarrow.filesystem.FileSystem``), From a0e2aa5eace81950c513dea9c9bf8ee0ea11d9f4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andrei=20B=C3=A2rsan?= Date: Thu, 13 May 2021 09:42:48 -0400 Subject: [PATCH 3/3] Replace warning with doc clarification --- docs/source/features.rst | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/source/features.rst b/docs/source/features.rst index 2138d3810..9d760a4e5 100644 --- a/docs/source/features.rst +++ b/docs/source/features.rst @@ -94,20 +94,20 @@ object. PyArrow integration ------------------- -.. warning:: - ``fsspec`` is only compatible with PyArrow's classic ``FileSystem``. Starting with version - 2.0 (released in October 2020), `pyarrow`_ switched to a C++ filesystem implementation that - is no longer compatible with ``fsspec``. Please refer to - `this GitHub issue `_ for additional details. - - -`pyarrow`_ has its own internal idea of what a file-system is (``pyarrow.filesystem.FileSystem``), +`pyarrow`_ has its own internal idea of what a file-system is (``pyarrow.fs.FileSystem``), and some functions, particularly the loading of parquet, require that the target be compatible. As it happens, the design of the file-system interface in ``pyarrow`` *is* compatible with ``fsspec`` -(this is not by accident). Therefore at import time, ``fsspec`` checks for the existence of -``pyarrow``, and, if found, adds it to the superclasses of the spec base-class. In this manner, -all ``fsspec``-derived file-systems are also pyarrow file-systems, and can be used by pyarrow -functions. +(this is not by accident). + +At import time, ``fsspec`` checks for the existence of ``pyarrow``, and, if ``pyarrow < 2.0`` is +found, adds its base filesystem to the superclasses of the spec base-class. +For ``pyarrow >= 2.0``, ``fsspec`` file systems can simply be passed to ``pyarrow`` functions +that expect ``pyarrow`` filesystems, and ``pyarrow`` `will automatically wrap them +`_. + +In this manner, all ``fsspec``-derived file-systems are also ``pyarrow`` file-systems, and can be used +by ``pyarrow`` functions. + .. _pyarrow: https://arrow.apache.org/docs/python/