Pathlib.iterdir semantics change dramatically under Python 3.13

[jaraco]

Bug report

Bug description:

This code behaves very differently on Python 3.13 than 3.12:

(p for p in pathlib.Path('does-not-exist').iterdir())

 🐚 py -3.12 -c "import pathlib; (p for p in pathlib.Path('does-not-exist').iterdir())"
 🐚 py -3.13 -c "import pathlib; (p for p in pathlib.Path('does-not-exist').iterdir())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import pathlib; (p for p in pathlib.Path('does-not-exist').iterdir())
                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pathlib/_local.py", line 575, in iterdir
    with os.scandir(root_dir) as scandir_it:
         ~~~~~~~~~~^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'does-not-exist'

In Python 3.12, the generator expression was evaluated lazily, not invoking path.iterdir() until the generator was consumed. On 3.13, at least a portion of the generator is evaluated, triggering the exception when the dir does not exist.

I discovered this issue in jaraco/jaraco.develop#26.

I looked at What's New for Python 3.13, and there's only one mention of generator expressions regarding mutation of locals in generator expressions, which doesn't seem to be relevant here.

That change mentions #74929, so maybe that change is also implicated in the change in execution order.

CPython versions tested on:

3.13

Operating systems tested on:

macOS

[jaraco]

Silly me. I just realized this issue probably isn't about a change to the generator expression but a change to pathlib.

[jaraco]

Indeed, the generator expression is not relevant:

 🐚 py -3.12 -c "import pathlib; pathlib.Path('does-not-exist').iterdir()"
 🐚 py -3.13 -c "import pathlib; pathlib.Path('does-not-exist').iterdir()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import pathlib; pathlib.Path('does-not-exist').iterdir()
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pathlib/_local.py", line 575, in iterdir
    with os.scandir(root_dir) as scandir_it:
         ~~~~~~~~~~^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'does-not-exist'

[tomasr8]

Bisected to #107320 so confirming that it's related to iterdir() and not the generator itself.

[jaraco]

I was even involved in that issue and had completely forgotten.

So it was an intentional change, even though it doesn't appear in the "what's new". At the very least, we should add a note to "what's new", as this behavior might be surprising.

I'm also thinking we should revisit our assumptions from #78722 in light of this new experience. It demonstrates that the change in behavior isn't necessarily beneficial and there may be other use-cases out there that are relying on the late evaluation of the error condition.

I don't feel strongly about it, and I'm happy to proceed with this new behavior, but we should ask ourselves if a rollback is warranted to retain consistency across Python versions.

[barneygale]

I'll get a PR up for adding it to the whatsnew, and perhaps to the Path.iterdir() docs

I still prefer the new behaviour - it meets most users expectations better, it makes exception handling much more straightforward, and it matches how os.scandir() raises exceptions too.

[cbornet]

This is indeed a big change.
Some libs such as anyio are relying on the fact that iterdir() doesn't do blocking syscalls and defer to a thread only when it's iterated.

[barneygale]

Maybe we should revert the iterdir() change then, despite it being a little awkward to handle exceptions previously.

[jaraco]

I'm so conflicted on this, I can't even advocate for one approach over the other. I'm inclined to say we should raise the issue with a larger group. I do think it would be worthwhile to make a decision quickly to reduce the duration of exposure. Maybe just a post in Core Dev discuss? I'm happy to defer to your judgment Barney, but let me know if you'd like my help garnering a wider consensus.