ArrowFSWrapper opens as seekable=False by default

[tdoehmen]

First, thanks for the great work! I tried using the ArrowFSWrapper to register Arrow filesystems with DuckDB's fsspec integration (https://duckdb.org/docs/guides/python/filesystems) as it would be a really cool way to access HDFS from DuckDB.

However, there were two issues that I encountered, that I want to bring to your attention.

The DuckDB fsspec integration assumes that, if the filesystem is capable of seeking, the file handle returned from .open() is seekable. For ArrowFSWrapped fsspec filesystems, however, seekable has to be set to explicitly upon calling .open(). Since seekable= is not a standard fsspec parameter, DuckDB is not aware of it and does not set it.

A solution on my side would be to write another wrapper around ArrowFSWrapper that overwrites the .open() behaviour, but I wonder
why the ArrowFSWrapped filesystems are non-seekable by default. IMO it would increase the consistency with other fsspec filesystems if seekable was True by default. Would you be open to accepting a PR that changes the default of ArrowFSWrapper
._open(seekable=) to True?

[martindurant]

I think seekable is not the default because it wasn't always available. I would be happy to change the default, but perhaps there are people depending on the current behaviour? I assume seekable is less performant, but are there other potential downsides?

if the filesystem is capable of seeking

I am wondering what this means

[tdoehmen]

if the filesystem is capable of seeking

What I mean is that some filesystems might inherently not support seeking to random places in the file (e.g. TarFilesystem could be an example?). But for HDFS, now that seeking is fixed in fsspec (since a couple of weeks? :)), why artificially limit this capability by default?

but perhaps there are people depending on the current behaviour?

I mean there is is only one way to find out... :P But I understand that point, and it's for sure something to be considered. If it's only the performance that would potentially be impacted, I think a change with a note in the relase notes would be justifiable, but that's only my personal take :)

[martindurant]

@efiop , perhaps an independent opinion on this?

[efiop]

Regarding arrowfs specifically, I agree that we should just switch the default to seekable open_input_file since it is natively supported by arrow and ticks more usability boxes than open_input_stream.

IIRC the reason why it wasn't seekable from the moment it was implemented is that we used to depend on an older HadoopFileSystem a few arrow versions ago that significantly differed from the new (current version) API. And then in #1091 it was just backward compatibility (good citizen behaviour, nothing wrong with that) to make it an opt-in flag. We should just switch to open_input_file by default and probably not even give the option to make it non-seekable. Where open_input_stream would be probably better is in get_file and cat_file (which is not implemented right now and so falls back to self.open() which is currently using open_input_stream), unless there is some native arrow method for downloading files that I'm not aware of.

Also seekable is a part of IOBase and some implementations (like local) already use it and that's the correct way to signal to the users that a file is seekable or not. So clearly default should be to implement seekable in any filesystem that supports it and in others to just make seekable return False.