Crash in urllib/request.py proxy_bypass_macosx_sysconf

[Tristan971]

Crash report

Python process crashes with a SIGABRT signal at the OS level (showing the standard software crash prompt of macOS, most of the time), and a python fatal error other times (the latter being shown here).

Unfortunately I don't have a minimal reproducer on hand.

However it looks exceptionally like a return of #72529 in the offending cause, as well as the workaround (setting the environment variable no_proxy to * fixes it too).

Which might share the same root cause (here: https://bugs.python.org/issue31818) at first glance.

Error messages

objc[79432]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[79432]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
Fatal Python error: Aborted

Current thread 0x00000001f66e1e00 (most recent call first):
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 2636 in proxy_bypass_macosx_sysconf
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 2660 in proxy_bypass
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/requests/utils.py", line 814 in should_bypass_proxies
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/requests/utils.py", line 830 in get_environ_proxies
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/requests/sessions.py", line 761 in merge_environment_settings
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/requests/sessions.py", line 579 in request
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/hvac/adapters.py", line 331 in request
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/hvac/adapters.py", line 372 in request
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/hvac/adapters.py", line 110 in get
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/hvac/api/auth_methods/token.py", line 278 in lookup_self
  File "/Users/tristan/.ansible/collections/ansible_collections/community/hashi_vault/plugins/module_utils/_auth_method_token.py", line 104 in authenticate
  File "/Users/tristan/.ansible/collections/ansible_collections/community/hashi_vault/plugins/module_utils/_authenticator.py", line 95 in authenticate
  File "/Users/tristan/.ansible/collections/ansible_collections/community/hashi_vault/plugins/lookup/hashi_vault.py", line 273 in run
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/template/__init__.py", line 1032 in _lookup
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/jinja2/runtime.py", line 290 in call
  File "<template>", line 13 in root
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/template/__init__.py", line 1156 in do_template
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/template/__init__.py", line 886 in template
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/template/__init__.py", line 930 in template
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/playbook/base.py", line 650 in post_validate
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/playbook/task.py", line 285 in post_validate
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/executor/task_executor.py", line 515 in _execute
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/executor/task_executor.py", line 158 in run
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/executor/process/worker.py", line 163 in _run
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/executor/process/worker.py", line 124 in run
  File "/Users/tristan/foo/bar/ansible/mitogen/ansible_mitogen/strategy.py", line 149 in <lambda>
  File "/Users/tristan/foo/bar/ansible/mitogen/mitogen/core.py", line 647 in _profile_hook
  File "/Users/tristan/foo/bar/ansible/mitogen/ansible_mitogen/strategy.py", line 148 in wrap_worker__run
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_fork.py", line 71 in _launch
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_fork.py", line 19 in __init__
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 281 in _Popen
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 121 in start
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/executor/process/worker.py", line 91 in start
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/plugins/strategy/__init__.py", line 393 in _queue_task
  File "/Users/tristan/foo/bar/ansible/mitogen/ansible_mitogen/strategy.py", line 291 in _queue_task
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/plugins/strategy/linear.py", line 315 in run
  File "/Users/tristan/foo/bar/ansible/mitogen/ansible_mitogen/strategy.py", line 321 in <lambda>
  File "/Users/tristan/foo/bar/ansible/mitogen/mitogen/core.py", line 647 in _profile_hook
  File "/Users/tristan/foo/bar/ansible/mitogen/ansible_mitogen/strategy.py", line 320 in run
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/executor/task_queue_manager.py", line 321 in run
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/executor/playbook_executor.py", line 190 in run
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/lib/python3.11/site-packages/ansible/cli/playbook.py", line 137 in run
  File "/Users/tristan/.local/share/virtualenvs/baz-8Eg-uX1t/bin/ansible-playbook", line 128 in <module>

Extension modules: markupsafe._speedups, yaml._yaml, _cffi_backend, charset_normalizer.md (total: 4)
ERROR! A worker was found in a dead state

Your environment

$ python -VV      
Python 3.11.4 (main, Jun  7 2023, 00:34:59) [Clang 14.0.3 (clang-1403.0.22.14.1)]

on macOS 13.4 on arm (M1)

Note that I had the same issue on 3.10.12 and on some version (that I can't recall) of 3.9 before; long meant to report the bug but hadn't yet bothered

Linked PRs

• gh-105912: document gotcha with using os.fork on macOS #112871
• [3.12] gh-105912: document gotcha with using os.fork on macOS (GH-112871) #113133
• [3.11] gh-105912: document gotcha with using os.fork on macOS (GH-112871) #113135

[Tristan971]

Not sure it'll be particularly useful, but just in case, here's the more generic software crash report generated by macOS in this case.

macos_report.txt

[ronaldoussoren]

This appears to be a duplicate of #104578 given the first few lines of the error message. That issue is still open because this issue needs to be documented somewhere.

The _scproxy extension can crash when used in a child proces started using fork without execv. That's a system issue on macOS.

[Tristan971]

This appears to be a duplicate of #104578 given the first few lines of the error message.

Indeed it seems like it is, sorry; didn't show up when I was searching the issue tracker somehow.

That's a system issue on macOS.

So I do see in that issue you quote, which is recent (May 2023), and the similar issue I linked (Oct 2016), and the one that one duped (Jan 2012 #58037), and on the 2016 mailing list exchange I also linked, and a few others that are now rather old...

Clearly this issue keeps occuring, and one of the only comments I can find officially suggesting a position at all from CPython on whether it's a defect was in #58037 (comment) in 2012, with the commenter suggesting that "it is not fixable in Python."

And I read the related blog post from Barry Warsaw on the matter. Very cool, but also quite puzzling that for a rather well-understood problem, we're still getting unexplained segfaults...

From what I see:

• os.fork without immediate exec* is rather unsafe on macOS 10.13 (chances of something non-trivial ending up calling a native library is not that low)
• Indeed, from various issues and comments inter-linked, it seems that many more native libraries are affected than just scproxy, as it is a general fork-safety policy change on Apple's side
• No one has a clear list of the libraries affected
• There's no plan to add safeguards on the Python side of it even for stdlib modules (and that's fair, tbh)

But if that's all correct, then it sounds a lot like rather than a note in the docs that no-one will notice, os.fork should straight up refuse to execute on macOS 10.13+ unless the user has set some kind of "YES_PLEASE_ALLOW_OS_FORK_EVEN_IF_IT_IS_LIKELY_TO_CRASH" envvar, and that no matter the macOS version. Otherwise a library maintainer running their tests/CI on another version has almost no chance of even being aware of the issue...

It is overall quite puzzling to have such an old and well-understood footgun dealt with with a "Python devs should just know to always exec right after fork for their stuff if it's to ever run on macOS; and if they don't, then oh well, they should have read the docs and paid close attention while doing so" approach...

That sort of expectation reminds me much more of relatively arcane pre-initialization expectations in lower-level languages than of something to expect from what is the de-facto managed runtime of the most popular scripting (and arguably generally programming) language. :/

[ronaldoussoren]

Removing os.fork on macOS is a couple of steps too far. It is possible to use os.fork without calling execv, but you have to be more careful about which APIs you use than on other platforms. In particular, all high-level APIs (e.g. anything defined in /System/Library/Frameworks is unsafe when using os.fork).

The proxy getting code just ends up in crash reports because that and tkinter are the only stdlib extension modules that use high-level APIs.

We cannot fix these crashes because it is platform behaviour we have no control over. And yes, that's annoying for us too.