pathlib.Path.iterdir doesn't raise an exception until you start iterating

[PaulPinterits]

BPO	34541
Nosy	@pitrou, @prudvinit
PRs	bpo-34541: pathlib.Path.iterdir throw an exception when path is not valid #8996 gh-78722: Fixed, pathlib.Path.iterdir now throws an exception when path is not valid #8999

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-08-29.08:20:21.120>
labels = ['3.8', 'type-feature', 'library']
title = "pathlib.Path.iterdir doesn't throw an exception until you start iterating"
updated_at = <Date 2019-05-23.22:47:36.119>
user = 'https://bugs.python.org/PaulPinterits'

bugs.python.org fields:

activity = <Date 2019-05-23.22:47:36.119>
actor = 'cheryl.sabella'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-08-29.08:20:21.120>
creator = 'Paul Pinterits'
dependencies = []
files = []
hgrepos = []
issue_num = 34541
keywords = ['patch']
message_count = 3.0
messages = ['324307', '324332', '324334']
nosy_count = 3.0
nosy_names = ['pitrou', 'Paul Pinterits', 'prudvinit']
pr_nums = ['8996', '8999']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue34541'
versions = ['Python 3.8']

Linked PRs

• GH-78722: Raise exceptions from pathlib.Path.iterdir() without delay. #107320

[PaulPinterits]

The fact that Path.iterdir() only throws exceptions once you start iterating over it makes it very difficult to write correct code.

Let's look at an example: We'll iterate over all children of a directory and print their file size.

If we try to do it like this, the try...except has no effect whatsoever:

try:
    children = path_that_doesnt_exist.iterdir()
except FileNotFoundError:
    print("directory doesn't exist")
for path in children:
    print(path.stat().st_size)

If we explicitly check whether the path exists (and is a directory), we end up with a race condition:

if path_that_doesnt_exist.is_dir():
    print("directory doesn't exist")
else:
    for path in children:
        print(path.stat().st_size)

We can wrap the whole loop in a try...except, but then we might end up catching exceptions we didn't intend to catch. (For example, the exception that's thrown when we try to get the size of a broken symlink.)

try:
    for path in path_that_doesnt_exist.iterdir():
        print(path.stat().st_size)  # this can also throw FileNotFoundError
except FileNotFoundError:
    print("directory doesn't exist")

We can manually call next on the iterator inside of a try..except block, but that's awfully verbose and requires knowledge about iterators and the next function:

children = iter(path_that_doesnt_exist.iterdir())
while True:
    try:
        path = next(children)
    except FileNotFoundError:
        print("directory doesn't exist")
        break
    print(path.stat().st_size)

Or we can turn the iterator into a list inside of a try...except, which seems to be the best option, but completely defeats the point of having an iterator:

try:
    children = list(path_that_doesnt_exist.iterdir())
except FileNotFoundError:
    print("directory doesn't exist")
else:
    for path in children:
        print(path.stat().st_size)

As you can see, writing correct (and good) code with iterdir is more difficult than it has any right to be. Please change this behavior so that exceptions are thrown immediately when iterdir is called.

[PaulPinterits]

As an afterthought, I'd like to suggest an alternative solution: If changing the iterdir behavior is not possible or not desirable for some reason, please add a Path.listdir method that returns a list instead of an iterator. Lazy file system operations can often introduce unnecessary race conditions in the code, so I believe it would be very useful if every lazy method had an eager equivalent.

[prudvinit]

Made changes, pathlib.Path('.').iterdir() now throws a FileNotFoundError if the path is not valid.

[barneygale]

Work-around: call list(path.iterdir()):

try:
    children = list(path_that_doesnt_exist.iterdir())
except FileNotFoundError:
    print("directory doesn't exist")
for path in children:
    print(path.stat().st_size)

I don't think we can change the iterdir() behaviour at this stage, nor does it justify the introduction of a similar method like listdir() to my mind.

[CAM-Gerlach]

Yeah, the fact that it returns a generator/lazy iterator seems pretty fundamental to iterdir's behavior, and matches the other current and past iter* functions/methods in Pythons, as well as pathlib.glob and similar methods (as well as builtins like range and others) which would make it suddenly behaving eagerly quite surprising to many/most users, as well as sacrificing performance/efficiency.

list(<lazy iterator>) to make something iterable is a common idiom in Python to handle this situation (and others), and adding a separate method to do the same thing eagerly seems thoroughly redundant.