subprocess raises misleading exceptions from child failure pre-exec

[geofft]

Bug report

If something goes wrong in subprocess after spawning the child process but before exec, subprocess assumes the only possible error can be from changing directory to the cwd=... argument and adds that path to the exception. I suppose this was correct at some point in the distant past, but there are many things that subprocess can do pre-exec.

The relevant code is this, from _execute_child in Lib/subprocess.py:

                if issubclass(child_exception_type, OSError) and hex_errno:
                    errno_num = int(hex_errno, 16)
                    child_exec_never_called = (err_msg == "noexec")
                    if child_exec_never_called:
                        err_msg = ""
                        # The error must be from chdir(cwd).
                        err_filename = cwd
                    else:
                        err_filename = orig_executable
                    if errno_num != 0:
                        err_msg = os.strerror(errno_num)
                    raise child_exception_type(errno_num, err_msg, err_filename)

That comment is wrong - there's now many things that can fail in the noexec phase in Modules/_posixsubprocess.c, all of which (besides preexec_fn) report errno to the parent as OSError. Here's a couple of sample misleading errors:

>>> subprocess.check_call(["pwd"], pass_fds=[30,], cwd="/tmp")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 368, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 349, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 9] Bad file descriptor: '/tmp'

>>> subprocess.check_call(["pwd"], user="root", cwd="/tmp")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 368, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 349, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 1] Operation not permitted: '/tmp'

(The same issue also happens with asyncio.create_subprocess_exec which uses subprocess.Popen under the hood.)

Your environment

Tested on macOS Ventura 13.3's /usr/bin/python3 (3.9.6) as well as docker run --rm -it python:3.12.0a7-bullseye.

Linked PRs

• gh-104522: Change message returned by subprocess when cwd option is not the cause #104851
• gh-104522: Fix OSError raised when run a subprocess #114195
• [3.12] gh-104522: Fix OSError raised when run a subprocess (GH-114195) #114219
• gh-104522: Fix test_subprocess failure when build Python in the root home directory #114236
• [3.12] gh-104522: Fix test_subprocess failure when build Python in the root home directory (GH-114236) #114239
• [3.11] gh-104522: Fix OSError raised when run a subprocess (GH-114195) #114243
• [3.11] gh-104522: Fix test_subprocess failure when build Python in the root home directory (GH-114236) #114245

[PurityLake]

If you aren't currently working on a fix for this, I have some time later to work on it. If you are let me know

[geofft]

I'm not working on this myself, but I'd love to see a fix!

[geofft]

As a personal opinion here - the change in #104851 looks right to me, but it relies a bit on the fact that nothing in Modules/_posixsubprocess.c's child_exec function, from the start until reached_preexec = 1, can cause either file-not-found or not-a-directory. This is true now, but might not be true in the future, and the fundamental cause of this bug is that it used to be true that the only thing that could fail was chdir and now there's more stuff happening. I think it would be good to add a comment in child_exec noting that the error is interpreted in this way and if you add new code you should make sure the errors continue to be interpreted right.

However, I wonder if it's possible to actually track the function that was called and report that back to the parent process, so the error is informative. (I ran into this in a production system that raised "Bad file descriptor," and I still haven't figured out which call raised it and why the file descriptor was bad.) This would require changing the format used by errpipe_write. I'm not sure if it's actually possible, since there's constraints here around memory allocation etc.

(Still, your fix is definitely an improvement and I'd like to see it merged....)

[PurityLake]

I will look into it and see what can be done