sage-download-file: Fix certificate problems with https downloads from upstream_url when sage-system-python is XCode's python3 on macOS

[mkoeppe]

#26351 added an optional upstream_url field to build/pkgs/*/checksums.ini. It streamlines the procedure for testing upgrade tickets: Developers or automatic testing facilities can pass an extra flag -o to sage-spkg to allow downloading from upstream rather than from Sage mirrors (where the updated ticket will be made available later only).

Many upstream package URLs use the https protocol - in contrast to the http protocol used when downloading from the Sage mirrors. The downloading is done via build/bin/sage-download-file, which uses the urllib module. It supports the https protocol.

However, SSL certificate problems are common on test systems. For example, if one uses XCode's python3 as the system python, then urllib does not automatically uses the standard system certificates. (This is apparently a known issue -- which is considered "wontfix" by Apple as reported here:
HandBrake/HandBrake#2216 (comment))

We add an option --no-check-certificate to sage-download-file, disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification).

Developers can set this option using the environment variable SAGE_DOWNLOAD_FILE_OPTIONS when installing packages (either by make or by using sage -i).

We note that even with SSL certificates disabled, there is still cryptographic protection because of the checksums recorded in checksums.ini.

---

Other possible workarounds considered:

• switching from using urllib directly to the requests library
• passing cafile=..., capath=... to urllib, perhaps coming from an environment variable
• using python instead of python3 on macOS as system python

As of #29090 (sage-system-python fixup) prefers /usr/bin/python3 over /usr/bin/python, leading to:

  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

(see https://github.com/mkoeppe/sage/runs/538432620)

CC: @vbraun @dimpase @kiwifb @jhpalmieri @videlec @fchapoton @kliem

Component: build

Author: Matthias Koeppe

Branch: 90ea00b

Reviewer: Jonathan Kliem

Issue created by migration from https://trac.sagemath.org/ticket/29418

[mkoeppe]

Description changed:

--- 
+++ 
@@ -2,3 +2,10 @@
 
 As seen with `local-homebrew-macos-minimal` in https://github.com/mkoeppe/sage/runs/538432620
 
+As of #29090 (sage-system-python fixup) prefers `/usr/bin/python3` over `/usr/bin/python`, leading to:
+
+```
+  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
+    self._sslobj.do_handshake()
+OSError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
+```

[mkoeppe]

comment:2

https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification

[mkoeppe]

comment:3

https://stackoverflow.com/questions/57630314/ssl-certificate-verify-failed-error-with-python3-on-macos-10-15

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,6 +1,13 @@
 This is for downloading from an https `upstream_url` when the host is poorly configured.
 
-As seen with `local-homebrew-macos-minimal` in https://github.com/mkoeppe/sage/runs/538432620
+In particular this happens with XCode's `python3` as the system python. This is apparently a known issue -- which is considered "wontfix" by Apple as reported here:
+https://github.com/HandBrake/HandBrake/issues/2216#issuecomment-527114519
+
+We should work around this by either:
+- switching from using urllib directly to the requests library
+- passing cafile=..., capath=... to urllib, perhaps coming from an environment variable
+- disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification), perhaps triggered by an option --no-check-certificate to sage-download-file
+- using python instead of python3 on macOS as system python
 
 As of #29090 (sage-system-python fixup) prefers `/usr/bin/python3` over `/usr/bin/python`, leading to:
 
@@ -9,3 +16,6 @@
     self._sslobj.do_handshake()
 OSError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
 ```
+(see https://github.com/mkoeppe/sage/runs/538432620)
+
+

[mkoeppe]

Branch: u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3

[dimpase]

comment:6

that's why I suggested not to bother with MacOS/Xcode's python(3).
Can't one get Python3 from python.org in MacOS by running a script?

---

New commits:

bd24641

build/bin/sage-download-file: Add option --no-check-certificate

[dimpase]

Commit: bd24641

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,12 +1,15 @@
-This is for downloading from an https `upstream_url` when the host is poorly configured.
+We add an option `--no-check-certificate` to `sage-download-file`, disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification).
+
+Users can set this option using the environment variable SAGE_DOWNLOAD_FILE_OPTIONS when using `sage -i`. This is for enabling them to download from an https `upstream_url` when the host is poorly configured.
 
 In particular this happens with XCode's `python3` as the system python. This is apparently a known issue -- which is considered "wontfix" by Apple as reported here:
 https://github.com/HandBrake/HandBrake/issues/2216#issuecomment-527114519
 
-We should work around this by either:
+---
+
+Other possible workarounds considered:
 - switching from using urllib directly to the requests library
 - passing cafile=..., capath=... to urllib, perhaps coming from an environment variable
-- disabling certificate checking (https://stackoverflow.com/questions/36600583/python-3-urllib-ignore-ssl-certificate-verification), perhaps triggered by an option --no-check-certificate to sage-download-file
 - using python instead of python3 on macOS as system python
 
 As of #29090 (sage-system-python fixup) prefers `/usr/bin/python3` over `/usr/bin/python`, leading to:

[mkoeppe]

Changed commit from bd24641 to none

[mkoeppe]

Changed branch from u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3 to none

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

comment:8

Replying to @dimpase:

that's why I suggested not to bother with MacOS/Xcode's python(3).
Can't one get Python3 from python.org in MacOS by running a script?

Please... the idea is to make Sage installation easier, not harder.

[mkoeppe]

comment:9

One often runs into these certificate errors also with random Linux images when I run them on Docker. So this workaround is valuable for the automatic testing in any case.

[mkoeppe]

Branch: u/mkoeppe/sage_download_file__fix_certificate_problems_with_https_downloads_on_macos_python3

[mkoeppe]

Commit: e143939