Cross-platform file system abstractions

[killercup]

(moderated summary by the WG)

• .., .., // not being auto- handled
• windows gotchas, like long file paths (which require extended-length paths aka "verbatim paths" as they're called in std)
  • https://github.com/dherman/verbatim
  • Automatically insert \\?\ where needed on Windows rust-lang/rust#32689
  • long file name support by default? BurntSushi/ripgrep#364
  • How do we deal with these and relative paths?
• absolute paths on Windows (UNC prefices)
  • https://github.com/notion-cli/notion/blob/master/crates/verbatim/src/lib.rs
• UTF-8 filenames on MacOS (the filesystem normalizes UTF-8 strings)
  • Glob patterns and umlauts on HFS vs. APFS BurntSushi/ripgrep#845
• atomic file writes
• Path.join: A method that both concatenates a path or completely replaces the existing one based on the input ... when did one ever have that use-case?
• "Simple" API that comes at the cost of performance for rapid prototyping / throwaway "scripts", see also https://github.com/rust-lang-nursery/cli-wg/issues/26
  • Priority should be on working out how the normal rust stuff should behave before wrapping it up in the high level, python-pathlib-like API.
• format! a path for a command line argument or mutating a path can cause problems with the non-UTF8 nature of OsStr

@killercup's original post

In the first meeting, we talked a bit about the pain points of dealing with files and path in a cross-platform manner.

One idea is to create or improve crates that provide higher-level abstractions than std in this area.

[killercup]

From the etherpad:

• file path handling
  • .., .., // not being auto- handled
  • windows gotchas, like long file paths (which require extended-length paths aka "verbatim paths" as they're called in std)
  • absolute paths on Windows (UNC prefices)
    • https://github.com/notion-cli/notion/blob/master/crates/verbatim/src/lib.rs
  • UTF-8 filenames on MacOS (the filesystem normalizes UTF-8 strings)

[killercup]

Maybe also relevant (also from the etherpad):

• streaming
  • how to thread progress computation through chained readers
  • handling different archive formats (this may be specific to me but I bet others will hit it)
    • how to stream zip files (no crates.io crate supports this AFAIK)
    • how to compute progress percentage of decompressing files

[matthiasbeyer]

I want to simply dump this here: I'm the author of imag and we have a rather nice abstraction for FS access... maybe you can learn from that. Our abstraction is rather specific for our use-case (each file has a TOML header and a "content" section), but the basic concept (a "store" which holds the files and they can then be borrowed from that store, so that concurrent access to one file is not possible) could be useful for other CLI apps.

More information about the Store can be found in its source or our (rather not-so-up-to-date, as we are basically a one-dev-project) documentation on the store. I'd say the source is a better source of information, though.

[sicking]

One thing that we talked about at Mozilla which would be useful in Firefox, as well as possibly useful to expose to the web, would be an API for doing atomic file writes in "small" files.

As I understood it, the best way to do truly atomic writes efficiently is to copy the file, while inserting the desired modifications, into a new file. Once the new file has been written issue a rename to the original file name.

Alternatively, if the file is small enough and multiple modifications are expected, read the file contents into memory and each time that a modification should be done write out a new file and rename to the desired file destination.

The nice thing with this approach is that it can be done without complex journaling files, and without needing to call flush() which can be quite a perf bottleneck.

This was something that we were thinking of using for the numerous small configuration/database files that Firefox keeps.

I think something like this would fit well with Rust's focus on performance and safety.

I'm not sure that this is particularly CLI specific though. But since other filesystem stuff was discussed I figured I'd mention it.

[luser]

windows gotchas, like long file paths (which require extended-length paths aka "verbatim paths" as they're called in std)

For some reason I thought the APIs in std did this internally but I was wrong. :( I believe Python's standard library converts all paths to verbatim paths internally nowadays. This is unfortunate in that it's really easy to run into the 256 character limit in practice, but I'm not sure there's a way to square this with the zero copy philosophy behind things like OsStr and Path.

AFAIK it's OK to use verbatim paths in all Windows APIs, so perhaps a crate that provides a small wrapper type would be good enough? Something like VPath { inner: Cow<Path> }, and then VPath::new would take AsRef<Path>, and if it has the \\?\ prefix borrow it, if not prepend it to an owned PathBuf. The real trick would be enforcing that all your APIs take this path type, so that you can ensure you're only ever passing verbatim paths to OS APIs. Without having it in std that's going to be a bit of a pain.

(If someone writes such a crate I would also like it to have an easy to_wide method for getting a wchar_t*-compatible buffer to pass directly to Windows APIs using ffi. I've found OsStrExt::encode_wide clunky to use in practice.)

[luser]

absolute paths on Windows (UNC prefices)
https://github.com/notion-cli/notion/blob/master/crates/verbatim/src/lib.rs

FYI this link 404s now, the code seems to have moved to https://github.com/dherman/verbatim

[luser]

FYI this link 404s now, the code seems to have moved to https://github.com/dherman/verbatim

...and after actually reading a little, it sounds like that crate is about 90% of what I proposed in my previous comment. :)

[BurntSushi]

See also:

• Automatically insert \\?\ where needed on Windows rust-lang/rust#32689
• long file name support by default? BurntSushi/ripgrep#364

I would be interested to here more about how other ecosystems solve this. In particular, how do they deal with relative paths?

[XAMPPRocky]

@killercup I believe normalisation on happens on HFS+ on macOS, I don't believe APFS does normalisation on the mac.

[BurntSushi]

On Unicode normalization and HFS: BurntSushi/ripgrep#845

[epage]

AFAIK it's OK to use verbatim paths in all Windows APIs, so perhaps a crate that provides a small wrapper type would be good enough?

My thought was to just start with a "normalize_path" crate/function that people can use regardless of what path API they are using, kind of like normalize-line-endings.

This crate would handle

• removing extraneous /, ., etc
• Resolve .. where possible (e.g. path/../file -> file)
• Convert to verbatim paths

Then on top of this we could look into an "easy paths" api that is more like python's pathlib combined with easy_strings to help with the prototyper case (easy to reach for needed utilities, less concern for borrow checker at cost of memory or cpu time). This would call "normalize_path" on any untrusted input. I've been providing feedback on ergo_fs in hopes of going in this direction.

In particular, how do they deal with relative paths?

What is your concern with relative paths?

[BurntSushi]

What is your concern with relative paths?

From MSDN:

Because you cannot use the "\\?\" prefix with a relative path, relative paths are always limited to a total of MAX_PATH characters.

[Screwtapello]

Resolve .. where possible (e.g. path/../file -> file)

This is not generally possible in POSIX because of symlinks: if a/b/c is a symlink to a/d, then a/b/c/../foo is actually a/foo, not a/b/foo. Now that modern versions of Windows support symlinks too, I suspect that's also an issue there.

That said, Rust kind of has this already in the form of std::fs::canonicalize(). It's still kind of clunky, though, because it requires every path component to exist. Compare the Python pathlib Path.resolve() method: "If strict is False, the path is resolved as far as possible and any remainder is appended without checking whether it exists." That means you can normalise your input and output paths while parsing your command-line, without having to create them first.