Doc: mention a minimal version of QEMU user emulation necessary for 3.13+?

[vfazio]

Documentation

I ran into an issue with running Python 3.13 under QEMU on Ubuntu 22.04.

>>> subprocess.run([sys.executable], check=True)
Traceback (most recent call last):
  File "<python-input-10>", line 1, in <module>
    subprocess.run([sys.executable], check=True)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/local/bin/python3']' returned non-zero exit status 127.
>>> subprocess._USE_POSIX_SPAWN = False
>>> subprocess.run([sys.executable], check=True)
Python 3.13.1 (main, Jan 21 2025, 16:37:53) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

Note that rc 127 is:

#define SPAWN_ERROR 127

I traced this down to 2b93f52 from #113118 which added support for using posix_spawn if posix_spawn_file_actions_addclosefrom_np exists.

The problem seems to be that while the libc may support posix_spawn_file_actions_addclosefrom_np, there needs to be equivalent support within the QEMU user emulator to support any syscalls made.

In this case, glibc uses the close_range syscall.

186898 Unknown syscall 436

On arm64, syscall 436 is close_range. (https://gpages.juszkiewicz.com.pl/syscalls-table/syscalls.html)

We'd expect:

27184 close_range(3,4294967295,0) = 0

glibc does try to fall back to closefrom, however that call can fail if the foreign chroot doesn't have /proc/ mounted, which is a situation often encountered when building foreign chroots without privilege escalation.

279077 openat(AT_FDCWD,"/proc/self/fd/",O_RDONLY|O_DIRECTORY) = -1 errno=2 (No such file or directory)
279077 exit_group(127)
 = 279077

close_range wasn't stubbed out in QEMU until 7.2 qemu/qemu@af804f39cc, meaning anyone running Ubuntu 22.04, Debian Bullseye, or potentially other non-EOL distributions are likely to run into problems unless they:

• deploy a hand built QEMU
• pull a newer QEMU package from a newer version of their distro
• upgrade their distribution

I don't think there's a great way to determine if the interpreter is running under user emulation, so Python likely can't handle this itself and avoid the posix_spawn call. The knob to disable posix_spawn, _USE_POSIX_SPAWN, requires manually flipping the flag which may not be possible when invoking scripts maintained by others (In my case, trying to install Poetry will call ensurepip which uses subprocess). There doesn't appear to be any environment variable knob for this either, so it's not as simple as exporting a variable when you know you're running a script in an emulated environment (though i'm not sure environment variable hand off is always guaranteed).

I'm wondering if there needs to be some documentation somewhere, either in subprocess or os.posix_spawn that calls out that users running Python under QEMU user emulation needs to have an emulator capable of handling this syscall, hopefully just to save them time debugging the issue.

I realize that making support guarantees under user emulation would be difficult, it's definitely not listed in the support tiers https://peps.python.org/pep-0011/ so understand if this gets closed without further discussion.

Tangentially related, os.closerange falls back to alternative solutions if close_range returns an error. The way Python has it wrapped, most cases will not use closefrom unless the caller specifies a rather large number for the high FD. This could cause problems for environments that don't have a mounted /proc since the libc implementation of closefrom raises an error if it can't query this directory.

@gpshead sorry to ping you but saw you looked over the original PR so figured I'd ask if you had thoughts on this?

Linked PRs

• gh-129204: Add _PYTHON_SUBPROCESS_USE_POSIX_SPAWN environment knob #132184
• [3.13] gh-129204: Add _PYTHON_SUBPROCESS_USE_POSIX_SPAWN environment knob (GH-132184) #132191

[vfazio]

docs may have been the wrong tag, but it's a place to start. I thought maybe documenting this would be good enough but maybe there needs to be further discussion per:

https://github.com/kulikjak/cpython/blob/1a577729e347714eb819fa3a3a00149406c24e5e/Doc/whatsnew/3.13.rst?plain=1#L1290-L1296

A private control knob :attr:`!subprocess._USE_POSIX_SPAWN` can
be set to ``False`` if you need to force :mod:`subprocess` not to ever use
:func:`os.posix_spawn`.  Please report your reason and platform details in
the CPython issue tracker if you set this so that we can improve our API
selection logic for everyone.

[vfazio]

For the sake of completeness, this is a:

x86_64 Ubuntu 22.04 host cross compiling/executing Python 3.13 in a aarch64 Debian Bookworm chroot.

[gpshead]

From a CPython perspective, qemu is not a supported platform as you've noted. That it gets used in emulation modes where it doesn't support various system calls that our Tier supported platforms do support is more of a qemu problem. I don't expect we're able to try to keep up with what qemu configurations available from whom can do what.

My previous employer used a qemu userspace target runtime for years while bringing up a new platform, they had to live with workarounds to things like this as a result until hardware usable in CI was sufficiently available. (ironically this motivated a colleague as contributor to help land our long desired vfork support as fork was either not supported or ultra-slow in this environment)

While I don't personally think qemu's odd shape is normally a justification for adding environment variable knobs to turn use of some features on or off. In the subprocess module case for things like posix_spawn or vfork or closefrom where we have existing boolean knobs settable at runtime as a failsafe to disable modern performant codepaths -

cpython/Lib/subprocess.py

Line 751 in 52ccf26

_USE_POSIX_SPAWN = _use_posix_spawn()

- it seems reasonable to add a set of _PYTHON_SUBPROCESS_XXX = (never|default|YOLO) environment variables that the code would check to override these defaults. I'd like to _ prefix those vars and, while documenting them is fine, state that they are internal implementation detail knobs that we reserve the right to remove in future python minor versions without a deprecation period as we only make a best effort claim that they do what they're intended to.

Ideally some lightweight logic within the use_posix_spawn() method could be worked out to detect the circumstances such as your qemu situation where it isn't usable if you have any suggested changes there.

... problems for environments that don't have a mounted /proc since the libc implementation of closefrom raises an error if it can't query this directory.

I expect a lot breaks in such an environment. we're already in a different code path in _posixsubprocess if /proc or equivalent doesn't exist to get a list of file descriptors from.

cpython/Modules/_posixsubprocess.c

Line 437 in 52ccf26

_close_range_except(start_fd, -1,

[vfazio]

Thanks for taking the time to respond.

The more I think about it, I think documenting this within CPython is a non-starter for all the reasons you mention, but I figured it wouldn't hurt to have a discussion.

As for a light weight check in use_posix_spawn, I'm not sure how best to do that in any intelligent way, at least not on the coffee i've had so far. The only thing that comes to mind is to return False if /proc/ isn't mounted, however that's highly dependent on the implementation of glibc, its fallback mechanisms, and the syscalls. This isn't always an indicator that we're running under emulation since it's perfectly fine to chroot into a root of the same architecture as the host and not have /proc mounted. However, not having proc means certain calls, like closefrom are almost certainly going to fail as those implementations in glibc (currently) rely on that mount.

I can try to putz around with the environment variable knob(s) if you're still willing to accept a PR for those since i have an environment where i can easily reproduce this situation. I'll probably just prioritize _PYTHON_SUBPROCESS_USE_POSIX_SPAWN since it's the master switch.

I think we'll have to reevaluate our CI to see what we can do about the mounts. The challenge is that we do not have ARM64 runners, and even if we did, the build times would likely be extremely slow. Cross compiling helps with the build speed, but since we're debootstraping a foreign architecture rootfs so we can generate disk images and SWUpdate payloads, there are certain steps that have to occur within the foreign root, like the installation of Python based applications. We manage dependencies of these applications with Poetry, so we have to install Poetry and install the application under emulation. Building on the ARM64 architecture would not alleviate the need to still chroot and run commands unfortunately, so the mount points are still an issue.

AFAIK, there is not a great way to cross build/package an application across architectures since it all depends on:

1. the version of Python being deployed
2. the architecture
3. the dependency list
4. if there are compatible wheels for the above combination
5. If there arent, then the build of those wheels
6. bundling everything into a venv or something similar to push to the foreign root

We currently don't have access to privileged containers, or containers with additional capabilities, such that we can make the mount points and subsequently mount -t proc /proc proc/ and other useful mountpoints. Default docker security doesn't allow unshare either, unless that's changed recently.

This particular build and setup process I MacGyver'd has worked for at least the past few years (with the exception of https://gitlab.com/qemu-project/qemu/-/commit/0266e8e3b3981b492e82be20bb97e8ed9792ed00) so I guess it was bound to break at some point.

I guess the other option we have, if i abuse this issue and think out loud, is to defer these steps until the board first boots and set up some services that run on first boot...

anyway, i'll get back with a PR or just out-right close this issue, so appreciate the back and forth.

[vfazio]

I have a quick "hack" that seems to work that is tri-state (1=Use posix_spawn, 0=Do not use posix_spawn, Unset or not present = default behavior):

diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index b2dcb1454c1..0038d3e22d3 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -711,6 +711,11 @@ def _use_posix_spawn():
     Prefer an implementation which can use vfork() in some cases for best
     performance.
     """
+
+    # Respect the environment variable override if present:
+    if (_val := os.environ.get('_PYTHON_SUBPROCESS_USE_POSIX_SPAWN')):
+        return bool(int(_val))
+

It works for the specific problem we've run into:

enter emulated chroot with affected QEMU

vfazio4 /home/vfazio/development/magicmissile :( # chroot output/rootfs/ bash
root@vfazio4:/# /usr/libexec/qemu-binfmt/aarch64-binfmt-P --version
qemu-aarch64 version 7.1.50 (v7.1.0-1640-g0d37413c63-dirty)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

posix_spawn enabled

root@vfazio4:/# export _PYTHON_SUBPROCESS_USE_POSIX_SPAWN=1

root@vfazio4:/# curl -sSL https://raw.githubusercontent.com/python-poetry/install.python-poetry.org/e518c5593346dc188e5ee174dacf50da3c78b826/install-poetry.py | POETRY_VERSION=1.8.5 python3 -
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/root/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.8.5): Restoring previously saved environment.
Poetry installation failed.
See /poetry-installer-error-v8u2_9kf.log for error logs.

posix_spawn default

root@vfazio4:/# export _PYTHON_SUBPROCESS_USE_POSIX_SPAWN=

root@vfazio4:/# curl -sSL https://raw.githubusercontent.com/python-poetry/install.python-poetry.org/e518c5593346dc188e5ee174dacf50da3c78b826/install-poetry.py | POETRY_VERSION=1.8.5 python3 -
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/root/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.8.5): Restoring previously saved environment.
Poetry installation failed.
See /poetry-installer-error-zppx_x8n.log for error logs.

posix_spawn disabled

root@vfazio4:/# export _PYTHON_SUBPROCESS_USE_POSIX_SPAWN=0

root@vfazio4:/# curl -sSL https://raw.githubusercontent.com/python-poetry/install.python-poetry.org/e518c5593346dc188e5ee174dacf50da3c78b826/install-poetry.py | POETRY_VERSION=1.8.5 python3 -
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/root/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.8.5): Done

Poetry (1.8.5) is installed now. Great!

To get started you need Poetry's bin directory (/root/.local/bin) in your `PATH`
environment variable.

Add `export PATH="/root/.local/bin:$PATH"` to your shell configuration file.

Alternatively, you can call Poetry explicitly with `/root/.local/bin/poetry`.

You can test that everything is set up by executing:

`poetry --version`