OSX 10.12 crash in urllib.request getproxies_macosx_sysconf and proxy_bypass_macosx_sysconf

[thehesiod]

BPO	28342
Nosy	@ronaldoussoren, @ned-deily, @thehesiod
Superseder	bpo-13829: exception error in _scproxy.so when called after fork
Files	python_urllib_crash.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-10-04.19:06:27.886>
created_at = <Date 2016-10-02.23:48:54.707>
labels = ['OS-mac', 'library']
title = 'OSX 10.12 crash in urllib.request getproxies_macosx_sysconf and proxy_bypass_macosx_sysconf'
updated_at = <Date 2016-10-04.19:06:27.872>
user = 'https://github.com/thehesiod'

bugs.python.org fields:

activity = <Date 2016-10-04.19:06:27.872>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2016-10-04.19:06:27.886>
closer = 'ned.deily'
components = ['Library (Lib)', 'macOS']
creation = <Date 2016-10-02.23:48:54.707>
creator = 'thehesiod'
dependencies = []
files = ['44940']
hgrepos = []
issue_num = 28342
keywords = []
message_count = 7.0
messages = ['277917', '277921', '277922', '277968', '277970', '278001', '278080']
nosy_count = 3.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'thehesiod']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '13829'
type = None
url = 'https://bugs.python.org/issue28342'
versions = ['Python 3.5']

[thehesiod]

I have a unittest which spawns several processes repeatedly. One of these subprocesses uses botocore, which itself uses the above two methods through the calls proxy_bypass and getproxies. It seems after re-spawning the methods a few times the titled calls eventually repeatedly cause python to crash on 10.12. I have a core file if that would help, zip'd it's ~242MB.

I've attached a file that shows the lldb callstack and python callstack.

[thehesiod]

interestingly I haven't been able to get this to crash in a separate test app. There must be either timing related to some interaction with another module. let me know how you guys would like to proceed. I can definitely reproduce it consistently in our application.

[ned-deily]

This is likely a duplicate of issues like bpo-27126 and bpo-13829. In bpo-24273 there is a suggested workaround: set environment variable "no_proxy" to "*".

http://bugs.python.org/issue24273#msg243963

[thehesiod]

ya I did a monkey patch which resolved it.

if sys.platform == 'darwin':
    import botocore.vendored.requests.utils, urllib.request
    botocore.vendored.requests.utils.proxy_bypass = urllib.request.proxy_bypass_environment
    botocore.vendored.requests.utils.getproxies = urllib.request.getproxies_environment

    urllib.request.proxy_bypass = urllib.request.proxy_bypass_environment
    urllib.request.getproxies = urllib.request.getproxies_environment

[ned-deily]

Glad you have it working. Did you try the suggested workaround of setting environment variable "no_proxy" to "*"? I believe that should have had the same net effect as the monkey patching.

[thehesiod]

I'm sure it would work, I just wanted a solution that didn't changes to our build infrastructure. btw if we're marking this as a duplicate of the other bug, can we update the other bug to say it affects python3.x as well? Thanks!

[ned-deily]

OK, I'm closing this a duplicate and I've updated bpo-13829 to include currently 3.x versions.