GH-127456: pathlib ABCs: add protocol for path parser

[barneygale]

Change the default value of PurePathBase.parser from ParserBase() to posixpath. As a result, user subclasses of PurePathBase and PathBase use POSIX path syntax by default, which is very often desirable.

Move pathlib._abc.ParserBase to pathlib._types.Parser, and convert it to a runtime-checkable protocol.

No user-facing changes as the pathlib ABCs are still private.

• Issue: pathlib ABCs: add protocols for supporting types #127456

[encukou]

Penny for your thoughts on this @encukou :)

This looks reasonable, but StatResult looks rather useless with its three mandatory but unreliable members:

• st_mode has UNIX permission bits and a platform-specific representation of file type (since AFAIK, POSIX doesn't specify the values of S_IFREG et.al.)
• st_ino & st_dev, which can be 0 if the info is not available -- if it weren't for the tuple API, the attributes could just as well be missing in this case.

Also, I guess typing has no way to say that e.g. if st_birthtime is present, it needs to be a time in seconds.

It seems that for your goal of supporting virtual filesystems, it might be better to rely on DirEntry, with stat remaining as a backend for filesystems where it's useful.
I wonder if we need a FS-agnostic version of st_ino/st_dev, like:

def key(self) -> Hashable:
    """Return a comparison key. Entries with the same key refer to the same file."""
    # For a Unix-like FS:
    ...
    return (stat_result.st_dev, stat_result.st_ino)

[barneygale]

Thank you for the feedback and reviews.

It seems that for your goal of supporting virtual filesystems, it might be better to rely on DirEntry, with stat remaining as a backend for filesystems where it's useful. I wonder if we need a FS-agnostic version of st_ino/st_dev, like:

def key(self) -> Hashable:
    """Return a comparison key. Entries with the same key refer to the same file."""
    # For a Unix-like FS:
    ...
    return (stat_result.st_dev, stat_result.st_ino)

I've occasionally thought about adding a PathBase.status() abstract method, which could return a DirEntry-like object with caching methods. Methods like PathBase.is_dir() could then be implemented via self.status().is_dir(). If we did that, we could either delete PathBase.stat(), or implement it in terms of self.status(). Would that be better?

I don't think DirEntry is an appropriate return type for status(), because we're not scanning the parent directory, rather we're querying the path directly. Perhaps it's just a naming problem.

[encukou]

I don't think DirEntry is an appropriate return type for status(), because we're not scanning the parent directory, rather we're querying the path directly. Perhaps it's just a naming problem.

Status, then? Like stat, but not a C-style abbr.
Or FileInfo?

[zooba]

I'm not a fan of the stat() model at all. It's the best of a bad lot, I know, but I'd rather have some kind of API that lets us use cached information for is_file()/is_dir()/etc. rather than having to guarantee that it's fresh for each call.

I'm not sure what such an API should look like, apart from I want it to have the Path object interface, and be using cached responses about the state of the path (or fetching them and caching on demand). There are more than enough properties that might as well be cached that it's worth it, and the TOCTOU issues basically all exist even with our current model.

[encukou]

Brainstorming:
Add a FileInfo protocol, implemented by classes that hold file info for a path. All the information in it is cached; the cache(s) might be initialized when an attribute is accessed (this can any attribute -- e.g. accessing the creation time might cache all of stat()), or when the FileInfo is created. The cache(s) can't be cleared (externally, FileInfo is immutable; but you can make a fresh instance).
Paths would get a cached_property-like info attribute, initialized to a FileInfo either on first access or when the Path is created (e.g. to hold extra info from iterdir).
As with cached_property, the cache can be cleared using del path.info.
We'd need a new_info() method as well, to hand out fresh info but leave the cache alone -- mainly as a backend for the current Path methods.

A downside is that this breaks the rule that touching the filesystem is done by methods, not attributes. (Can we say rule was for getting “fresh” information, and we can use something else here?)

[barneygale]

This all sounds great to me.

A downside is that this breaks the rule that touching the filesystem is done by methods, not attributes.

Why would accessing Path.info touch the filesystem? Couldn't make it so that only FileInfo.is_*() methods can perform FS access?

[barneygale]

Oh, I suppose you might want to lstat() or stat() so that FileInfo always represents something that exists/existed. Otherwise we'd need a FileInfo.exists() method, and if we're intending to store os.DirEntry objects as Path.info when paths are generated from iterdir(), then we'd need to add an os.DirEntry.exists() method too.

[barneygale]

Courtesy CCing @ncoghlan, who proposed something similar here. I shot it down then because it wasn't clear to me when to call [l]stat(), how to clear caches, and how it interacts with Path.is_*(). I'm also a tiny bit uneasy making Path stateful.

[zooba]

(Process note - we should probably move the API design discussion off this PR)

exists() shouldn't be an issue, because we can cache that as True or False at creation time (e.g. returned from glob() implies that exists() == True at some point in the recent past; created by a user implies that exists() is unknown and needs to make a filesystem call). The same could potentially be true for any other state, most obviously is_dir() and is_file(), but also any of the regular stat attributes (where a subclass could well return their own type from .stat()).

What we need to design is the way for a user to discover that they've got cached values, and reset it. A CachedInfo mixin (just for type check purposes) and a dedicated subclass (so that e.g. Path(p) would recreate and uncache all members) would seem to suffice, though I can also see the reasoning for adding a new member to manage it.

[barneygale]

Following up on the forum: https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/30

[barneygale]

I've removed DirEntry and StatResult from this PR, so it now covers only only Parser.

[zooba]

The reduced PR to just the parser changes LGTM

[barneygale]

Thanks both, very much appreciate the feedback :-)