certificate chain verification bug in esp-idf

[jerryneedell]

the following request works OK with and ESP32 with the NINA firmware, but not with the esp32s2 native wifi

response = requests.get("https://api.thingspeak.com/channels/1417/feeds.json?results=1")
 fails with 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "wifi_cheer.py", line 39, in <module>
  File "adafruit_requests.py", line 507, in get
  File "adafruit_requests.py", line 456, in request
  File "adafruit_requests.py", line 405, in _get_socket
  File "adafruit_requests.py", line 401, in _get_socket
OSError: Failed SSL handshake
>>> 

[jerryneedell]

If you can provide some guidance on how/where this needs to be done, I'll be happy to try.

[tannewt]

These are the API docs where I would start: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/esp_crt_bundle.html

@brentru might have experience with it for nina.

[jerryneedell]

Slowly trying to make some sense of this
the way certificates are configured in the sdconfig is quite different for the
ESP32S2
https://github.com/adafruit/circuitpython/blob/main/ports/esp32s2/esp-idf-config/sdkconfig.defaults#L548
vs
NINA
https://github.com/adafruit/nina-fw/blob/master/sdkconfig#L563

It is also not clear to me how to update the certificates used by the ids in the esp32s2
in the Docs it says the certificates used are quite old - Jan 2019 -- and the ones in the NINA build are much newer

The bundle comes with the complete list of root certificates from Mozilla’s NSS root certificate store. Using the gen_crt_bundle.py python utility the certificates’ subject name and public key are stored in a file and embedded in the ESP32 binary.

When generating the bundle you may choose between:

The full root certificate bundle from Mozilla, containing more than 130 certificates. The current bundle was updated Wed Jan 23 04:12:09 2019 GMT.
A pre-selected filter list of the name of the most commonly used root certificates, reducing the amount of certificates to around 35 while still having around 90 % coverage according to market share statistics.
In addition it is possible to specify a path to a certificate file or a directory containing certificates which then will be added to the generated bundle.

It also indicates that one can specify a file when building so I think that is the route to examine ...

[askpatrickw]

I looked into this last week for a hot second and while I could generate the updated bundle

/code/github/circuitpython/ports/esp32s2/esp-idf/components/mbedtls/esp_crt_bundle
python gen_crt_bundle.py --input cacrt_all.pem

The build didn't seem to act any different despite the build flags being ON to use the bundle.

/circuitpython/ports/esp32s2/esp-idf-config/sdkconfig.defaults

#
# Certificate Bundle
#
CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y
CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_FULL=y
# CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_CMN is not set
# CONFIG_MBEDTLS_CERTIFICATE_BUNDLE_DEFAULT_NONE is not set
# CONFIG_MBEDTLS_CUSTOM_CERTIFICATE_BUNDLE is not set
# end of Certificate Bundle

Not an answer, but maybe that points you in the right direction

[jerryneedell]

what version of cacrt_all.pem did you use?

[jerryneedell]

Or I should have asked, what process did you use to get the updated certificate files.

[askpatrickw]

I ran that python script from this folder circuitpython/ports/esp32s2/esp-idf/components/mbedtls/esp_crt_bundle

esp-idf/components/mbedtls/esp_crt_bundle on  master via 🐍 system took 2s
➜ python gen_crt_bundle.py --input cacrt_all.pem


gen_crt_bundle.py: Parsing certificates from cacrt_all.pem
gen_crt_bundle.py: Successfully added 135 certificates
gen_crt_bundle.py: Successfully added 135 certificates in total

And then you end up with a new x509_crt_bundle

➜ gst
HEAD detached at 8bc19ba89
Untracked files:
  (use "git add <file>..." to include in what will be committed)

	x509_crt_bundle

[jerryneedell]

Looking over the build process, it looks like the certificates are being updated as part of the build process
so I'm to sure this is the actual cause of the SSL handshake errors

If you look in the board build folder after completing a build, this is a file named x509_crt_bundle.S generated and I think this contain the certificate updates from the process noted above. It is run for every build.

[tannewt]

@jerryneedell How does the list of root certs compare to those in nina-fw? The connection issues might actually be due to the clock not being set. Checking the debug logging may provide more insight.

[anecdata]

This looks site-specific, so I think cert-related. I had never encountered the SSL handshake issue with my own server (HTTPS/TLS), but I can replicate with the link above. Doesn't matter if the RTC is 2000 or 2020.

Saola 1 w/Wrover with ESP32S2
6.0.0-rc.0 on 2020-10-16
adafruit-circuitpython-bundle-6.x-mpy-20201101
Requests 1.7.2