Some pre-finalization callbacks can create other callbacks

[ZeroIntensity]

Bug report

Bug description:

Currently, "pre-finalization" (callbacks executed while the interpreter is still fully intact) looks like this:

cpython/Python/pylifecycle.c

Lines 2023 to 2038 in 642e5df

	wait_for_thread_shutdown(tstate);
	// Make any remaining pending calls.
	_Py_FinishPendingCalls(tstate);
	/* The interpreter is still entirely intact at this point, and the
	* exit funcs may be relying on that. In particular, if some thread
	* or exit func is still waiting to do an import, the import machinery
	* expects Py_IsInitialized() to return true. So don't say the
	* runtime is uninitialized until after the exit funcs have run.
	* Note that Threading.py uses an exit func to do a join on all the
	* threads created thru it, so this also protects pending imports in
	* the threads created via Threading.
	*/
	_PyAtExit_Call(tstate->interp);

Threads are joined first, then pending calls are executed, and finally atexit callbacks are executed. The problem is that any of these three can create one another, such as a pending call creating a thread, or an atexit handler scheduling a pending call. The best way to demonstrate this is by creating a new thread inside of an atexit callback:

import atexit
import threading
import time


def run():
    print(24)
    time.sleep(1)
    print(42)

@atexit.register
def start_thread():
    threading.Thread(target=run).start()

On 3.13+ (and probably earlier), you'll get one of these three outputs:

1. 24 is printed.
2. Nothing is printed.
3. A fatal error occurs due to the stdout lock being unavailable.

This is because the thread created is incorrectly non-daemon, because the call to threading._shutdown (which should join it) has already happened.

@ericsnowcurrently, I discussed this with you at the PyCon sprints. I think we came to the agreement that the best way to fix this is by looping until all three of them execute no callbacks.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-136003: Execute pre-finalization callbacks in a loop #136004

[ericsnowcurrently]

Yeah, this is what we discussed. Thanks for remembering and opening this issue. I have some thoughts, but first I'll take a look at your PR.