Handle failure from image converter service or other HttpErrors in `wget()`

[dhalbert]

If wget() in PortalBase (https://github.com/adafruit/Adafruit_CircuitPython_PortalBase/blob/7a3277af2efb804c7ecf82d8bc87eb95c22dee2e/adafruit_portalbase/network.py#L319) gets some kind of HTTP error, it will raise adafruit_portalbase.network.HttpError, which is a subclass of Exception`.

This exception is not handled when wget() is called here:

Adafruit_CircuitPython_PyPortal/adafruit_pyportal/network.py

Lines 191 to 196 in a14d41b

	self.wget(image_url, filename, chunk_size=chunk_size)
	except OSError as error:
	raise OSError(
	"""\n\nNo writable filesystem found for saving datastream. Insert an SD card or set internal filesystem to be unsafe by setting 'disable_concurrent_write_protection' in the mount options in boot.py""" # pylint: disable=line-too-long
	) from error
	except RuntimeError as error:

.

A scenario where this happened is described in the forums here: the Cleveland Museum of Art may return image URL's that return 404's. These cause the image converter service to return a 422.

I'm thinking that maybe PortalBase should return an easier-to-catch exception, or PyPortal should catch that HttpError and turn it into something else.

[anglerfish27]

Hello brilliant minds!
I am just checking in to see if there are any updates on this. Doesn't look like it yet.
Any work around I could put in this file for the time being?

[anglerfish27]

Any luck. The CMA project code has gone from bad to worse. Now images don't load at all.
Having made no changes to the project. Still on 8.2.6 with 1.7.5 firmware for WiFi. Now I get the following when I plug in the Titano. I had to step away from this project for a bit, hence the delay as I was dealing with a family emergency.

Here's what I get now and I just the pre-stored CMA image that came with the project and it just sits there.

>>> %Run -c $EDITOR_CONTENT '`'
retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=1381
Connecting to AP MartyMcW-Fi
Retrieving data...An error occured, retrying! - Sending request failed
loop counter: 0
Traceback (most recent call last):
  File "adafruit_requests.py", line 534, in _get_socket
  File "adafruit_requests.py", line 764, in connect
  File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 75, in connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 806, in socket_connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 702, in socket_open
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 332, in _send_command_get_response
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 288, in _wait_response_cmd
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 197, in _wait_for_ready
TimeoutError: ESP32 not responding

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 51, in <module>
  File "/lib/adafruit_pyportal/__init__.py", line 305, in fetch
  File "adafruit_portalbase/network.py", line 518, in fetch
  File "adafruit_requests.py", line 823, in get
  File "adafruit_requests.py", line 668, in request
  File "adafruit_requests.py", line 515, in _get_socket
RuntimeError: Sending request failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 61, in <module>
KeyboardInterrupt: 
>>> 
Connection lost -- read failed: [Errno 6] Device not configured

Use Stop/Restart to reconnect.

Process ended with exit code 1.

[cogliano]

I haven't looked at this project in a while. I'll investigate and see what's going on.

--- Dan Cogliano

On 2023-10-17 11:11, anglerfish27 wrote: Any luck. The CMA project code has gone from bad to worse. Now images don't load at all. Having made no changes to the project. Still on 8.2.6 with 1.7.5 firmware for WiFi. Now I get the following when I plug in the Titano. I had to step away from this project for a bit, hence the delay as I was dealing with a family emergency. Here's what I get now and I just the pre-stored CMA image that came with the project and it just sits there. >>> %Run -c $EDITOR_CONTENT '`' retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=1381 Connecting to AP MartyMcW-Fi Retrieving data...An error occured, retrying! - Sending request failed loop counter: 0 Traceback (most recent call last): File "adafruit_requests.py", line 534, in _get_socket File "adafruit_requests.py", line 764, in connect File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 75, in connect File "adafruit_esp32spi/adafruit_esp32spi.py", line 806, in socket_connect File "adafruit_esp32spi/adafruit_esp32spi.py", line 702, in socket_open File "adafruit_esp32spi/adafruit_esp32spi.py", line 332, in _send_command_get_response File "adafruit_esp32spi/adafruit_esp32spi.py", line 288, in _wait_response_cmd File "adafruit_esp32spi/adafruit_esp32spi.py", line 197, in _wait_for_ready TimeoutError: ESP32 not responding The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<stdin>", line 51, in <module> File "/lib/adafruit_pyportal/__init__.py", line 305, in fetch File "adafruit_portalbase/network.py", line 518, in fetch File "adafruit_requests.py", line 823, in get File "adafruit_requests.py", line 668, in request File "adafruit_requests.py", line 515, in _get_socket RuntimeError: Sending request failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 61, in <module> KeyboardInterrupt: >>> Connection lost -- read failed: [Errno 6] Device not configured Use Stop/Restart to reconnect. Process ended with exit code 1. -- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

Links: ------ [1] #128 (comment) [2] https://github.com/notifications/unsubscribe-auth/ACUA3ESCYNKFQU6WONVEDN3X72NYXAVCNFSM6AAAAAA5FFLXYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRWGYZDAOBYGY

[anglerfish27]

Thank you @cogliano!

[cogliano]

So I reset my Titano, updating it to Circuitpython 8, removed all the files and installed the library bundle from the Learn guide. Verified the secrets.py I had previously and it all works for me. Maybe try these steps to see if it works for you. Are you using Circuitpython 8? Dan Cogliano

…

On Oct 17, 2023 at 11:23 AM, <anglerfish27 ***@***.***)> wrote: Thank you @cogliano (https://github.com/cogliano)! — Reply to this email directly, view it on GitHub (#128 (comment)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/ACUA3EQRWVNAZLPUOLAYDULX72PIJAVCNFSM6AAAAAA5FFLXYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRWGY2DKNJZGM). You are receiving this because you were mentioned.Message ID: ***@***.***>

[anglerfish27]

Thank you @cogliano for taking a look. I know you were the creator of this project and I really appreciate your help. I gave it another try today and did a full nuke and pave.

I loaded CP 8.2.7 for the Titano. I nuked the SPI flash to ensure to no gremlins were left over. Firmware of the WIFI is updated as well to 1.7.5

I am still facing this "new" 2nd issue. The first issue was is that it would load the images but a random point it would encounter a URL that returned a 404 (you could manually click it to confirm, examples are in the Adafruit forums. While troubleshooting that I put this project aside due to some family issues. I hadn't touched a thing. All I did was turn it on. and now I'm presented with this "2nd" issue that essentially bricks the program.

I'm going to lay out every thing I did today so you can understand 100% my status.

Brand new Titano (well maybe 3 months old?). Like I said I nuked the flash and installed CP 8.2.7 (before it was 8.2.6 and before that I was given a link to 6.x to try!). I have the latest WiFi drivers for the ESP32 1.7.5.

Folder structure on the Titano:
/lib
/lib/adafruit_bitmap_font
/lib/adafruit_display_shapes
/lib/adafruit_io
/lib/adafruit_minimqtt
/lib/adafruit_pyportal
/lib/adafruit_touchscreen.mpy
/fonts/OpenSans-9.bdf

/background.bmp
/background_480.bmp
/boot_out.txt
/code.py
/secrets.py

I have tested this very set of files and successfully executed the NASA image of the day project, along with the Adafruit quote's project. So it is connecting just fine to Wifi and the SD card is working (I have tried so many SD cards of varying size and manufacturer's (FAT32 of course). I tried using both the different API URLs provided. No difference. My Adafruit API key is valid in my secrets file which is proved by the fact the other projects mentioned use it and work fine.

I just tried the code.py from github as well on the adafruit website. No difference.

code.py

import time
import random
import board
from adafruit_pyportal import PyPortal
from adafruit_display_shapes.circle import Circle

WIDTH = board.DISPLAY.width
HEIGHT = board.DISPLAY.height

#pylint: disable=line-too-long

# these lines show the entire collection
APIURL = "https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&skip="
IMAGECOUNT = 31954

# uncomment these lines to show just paintings
#APIURL = "https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip="
# IMAGECOUNT = 3223

BACKGROUND_FILE = "/background.bmp"
if WIDTH > 320:
    BACKGROUND_FILE = "/background_480.bmp"

pyportal = PyPortal(default_bg=BACKGROUND_FILE,
                    image_json_path=["data", 0, "images", "web", "url"],
                    image_dim_json_path=(["data", 0, "images", "web", "width"],
                                         ["data", 0, "images", "web", "height"]),
                    image_resize=(WIDTH, HEIGHT - 15),
                    image_position=(0, 0),
                    text_font="/fonts/OpenSans-9.bdf",
                    json_path=["data", 0, "title"],
                    text_position=(4, HEIGHT - 9),
                    text_color=0xFFFFFF)

circle = Circle(WIDTH - 8, HEIGHT - 7, 5, fill=0)
pyportal.splash.append(circle)
loopcount = 0
errorcount = 0
while True:
    response = None
    try:
        circle.fill = 0xFF0000
        itemid = random.randint(1, IMAGECOUNT)
        # itemid = 20 # portrait mode example
        # itemid = 21 # landscape mode example
        print("retrieving url:", APIURL + str(itemid))
        response = pyportal.fetch(APIURL + str(itemid))
        circle.fill = 0
        print("Response is", response)
        loopcount = loopcount + 1

    except (RuntimeError, KeyError, TypeError) as e:
        print("An error occured, retrying! -", e)
        print("loop counter:", loopcount)
        assert errorcount < 20, "Too many errors, stopping"
        errorcount = errorcount + 1
        time.sleep(60)
        continue

    errorcount = 0
    stamp = time.monotonic()
    # wait 5 minutes before getting again
    while (time.monotonic() - stamp) < (5*60):
        # or, if they touch the screen, fetch immediately!
        if pyportal.touchscreen.touch_point:
            break

Upon saving and applying power to the Titano I get the CMA 480.bmp image displayed. That's where it sits there forever never progressing. Thonnny shows...:

retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=18017 Connecting to AP MartyMcW-Fi Retrieving data...An error occured, retrying! - Sending request failed loop counter: 0``

It will continue this error count until I manually control+C to break it, when I do Thonny throws the following:

`>>> %Run -c $EDITOR_CONTENT '`'
retrieving url: https://openaccess-api.clevelandart.org/api/artworks?cc0=1&has_image=1&indent=2&limit=1&type=Painting&skip=18017
Connecting to AP MartyMcW-Fi
Retrieving data...An error occured, retrying! - Sending request failed
loop counter: 0

Traceback (most recent call last):
  File "adafruit_requests.py", line 534, in _get_socket
  File "adafruit_requests.py", line 764, in connect
  File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 75, in connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 806, in socket_connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 702, in socket_open
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 332, in _send_command_get_response
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 288, in _wait_response_cmd
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 197, in _wait_for_ready
TimeoutError: ESP32 not responding

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 51, in <module>
  File "adafruit_pyportal/__init__.py", line 306, in fetch
  File "adafruit_portalbase/network.py", line 518, in fetch
  File "adafruit_requests.py", line 823, in get
  File "adafruit_requests.py", line 668, in request
  File "adafruit_requests.py", line 515, in _get_socket
RuntimeError: Sending request failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 61, in <module>
KeyboardInterrupt: `

I'm willing to do whatever is needed. I was really happy when it was working (so I thought) until it started hitting 404 URLs when it hit one it would freeze on whatever the last image displayed was. I bought the Titano for this project. I'm trying here I really am. I don't know what I'm doing wrong. Maybe your code is different than mine? I dunnno. $50+shipping and I can't even run the project :(

I have the smaller version of the Titano the PyPortal (regular) which is running the OpenWeather API project on Adafruit's site. That one works awesome too. It does crash every few days on socket issues. I'm going to try upgrading the code on it and see if I can debug it myself too. I don't mind having to push the reset button a few times every few days.

When you said it "works for you" how much time did you let it run? To call it stable you would probably need to let it run at least 3-4 days 24/7. Or maybe you fell into the trap I did and thought it was working great until I noticed images were not changing after awhile leading to this ticket and the whole 404 image issue.

Please help I really want to get this project working and stable. I have a nice picture frame I cut out a nice matted layer square to place it in and hide everything behind it so I can hang it on a wall. Kinda need the thing to work though :)

Thank you for your help and I am standing by on next steps..
Respectfully,
Anglerfish27

[anglerfish27]

If I wait long enough the LCD will display the following: (shortened but keeping the key points)

esp32spi.py line 197 in _wait_for_ready

The above exceptio was the direct cause of the following exception:
code.py line 51
/pyportal/__init__.py line 306 in fetch
/portalbase/network.py 518 inn fetch
adafruit_requests.py line 823 in get
adafruit_requests.py line 668 in request
adafruit_requests.py line 515 in _get_socket
Runtime error sending request failed.
Trace back last call line in code.py line 59
AssertationError Too many errors stopping.

[FoamyGuy]

@anglerfish27 I have been doing lots of testing around this issue and the cleveland art pyportal project to try to understand the issues you're facing.

It looks like you've faced a few different symptoms and I don't have answers or information about all of them, but I can try to help get you back to a state where the project can work.

In your second most recent message you had this error message:

Traceback (most recent call last):
  File "adafruit_requests.py", line 534, in _get_socket
  File "adafruit_requests.py", line 764, in connect
  File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 75, in connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 806, in socket_connect
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 702, in socket_open
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 332, in _send_command_get_response
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 288, in _wait_response_cmd
  File "adafruit_esp32spi/adafruit_esp32spi.py", line 197, in _wait_for_ready
TimeoutError: ESP32 not responding

I believe the root cause of this error is: adafruit/nina-fw#58

Unfortunately there is some underlying issue within nina-firmware (which is used by the ESP32 on the pyportal) that results in this error occuring when attempting to communicate with a server that uses a certain type of HTTPS certificate. I confirmed with Dan (the one who dug into this when it was noticed) that the cleveland art API is using a certificate of the type that there are problems with.

However I know that this project was working at one point in time, you had it running on your device for a while until some unrelated 404 issues popped up. I have run it on my devices at various times as well.

It turns out that the nina firmware that was pre-loaded on the ESP32 that's on the pyportal does not seem to have the same problem. So that "out of the box" firmware can successfully make requests and get responses from the cleveland art API. It proved to be a bit of a challenge to track down a copy of the bin file for that original firmware, but Dan helped with that and was able to pull a copy from a device he has.

I've loaded this bin on my pyportal Titano and tested it successfully with the cleveland art project. My device has circuitpython 8.2.7 loaded on it and the ESP32 has this specific firmware loaded and it's allowing the project to work.

Link to .bin file:
https://cdn.discordapp.com/attachments/1166371166806622210/1166531169383350344/nina-fw-122-from-titano.bin?ex=654ad3cf&is=65385ecf&hm=028f7970b16d3ca3a40754afc40cf3a0ddb3551ebc9b1f2d9848a004284e1066&

It looks like you're already familiar with the process of upgrading firmwares on the ESP32 because you mentioned loading the latest one. The process to load this one should be the exact same, just choose this bin file instead of any others.

Ordinarily upgrading the firmware would be the thing you want to do, but in this case unfortunately there was an issue that came up at some point with this type of certificate and there isn't currently a fix for it in any newer versions of the firmware. So right now the best work around to get the art project running is to use this version of the esp32 firmware for the co-processor.

I read through the thread here, on the adafruit forums, and in the issue you made on the cleveland art repo. It sounds like the initially reported 404 issues may have been alleviated on their side somehow. But I do also think it's possible for us to have better error handling for that in the project code that can have it choose a new number and try loading a new artwork. I'm going to leave my device running the art project for a while to see if I encounter any further issues with it. If I do I'll report back with information about them and hopefully potential fixes.

Please let me know if you have further issues or need help with the process to get this firmware version loaded. I am willing to try to help you get your pyportal back to running the art project, feel free to ping me here.

[anglerfish27]

Wow thank you @FoamyGuy. I'm speechless honestly. This level of effort from everyone involved is really giving me hope for humanity!! :) Thank you for the deep dive explanation. I am going to give the older WiFi a try and hopefully it works. Any hopes or ETA on a 1.7.6 FW from ExpressIF that will have ''all the things"? lol I know you don't control that.

Yeah the 404's where interesting until I tracked down what was happening at least within what I could see (I'm not super MP smart, just enough to poke at MP' libraries if needed). I opened a ticket with the CMA GitHub team as well around the 404's which they say should be fixed. I wonder if its of any value mentioning the issue you found with their cert type to them there? I mean the API is mostly going to be used by WiFi enabled devices, I doubt a lot of people use it on their computer so I would opine there's a good chance an ESP32 is involved and hindering their project from being more friendly to devices. Granted there are many other wifi modules. I dunno I'm thinking like my sysadmin days. Cert replacing can take 5 minutes or implode your entire environment if you don't know what you are doing!

Granted if CMA is doing something very odd with certs (very curious what you found in the certs as strange) maybe they would be thankful to know about it. Just a thought. The Github to CMA is https://github.com/ClevelandMuseumArt/openaccess/issues/8#issuecomment-1766889912.

Alright lets hope I don't brick this thing with the firmware downgrade! Going to get started on it now. (fingers crossed)

[dhalbert]

There isn't anything wrong with CMA's cert chain: they are using a Let's Encrypt cert chain with an ECDSA cert (elliptical). For some reason, NINA-FW has stopped handling these properly, sometime after 1.2.2. These are going to become more and more common, so we would like to make them work.

The proper compile options appear to be turned on in 1.7.x, so we are unsure what is happening. It is not a question of a missing root cert. We are going to compare the sources for 1.2.2 and 1.7.x and see if we can discern anything.

[anglerfish27]

Well good news team! CMA project is up and running on the NINA-FW 1.2.2 with CP 8.2.6!

It's been running for awhile now shortly after my post as it didn't take long to flash everything.
Thanks for the details about the cert, I figured there was nothing wrong with their cert since its valid, but I was curious what part of it (now I know the elliptical) part is what breaks the NINA-FW.

I'm going to let it continue to bake for the next few days and see how it holds up or if it jams up or not.

I may be opening a separate case for the regular Pyportal and the open weather project. I keep getting socket failures every few days. I know that has 1.7.5 for NINA-FW as well. But its a few revs back on CP. Maybe that will help. Anyways that's not for this case ticket/forum.

SO FAR SO GOOD!! I think if by tomorrow its still working we have fully root caused it.
-Hoping the enhanced error handling of the 404's is added to future code along with a new firmware that deals with these certs properly but I know that isn't on you guys so much.

Recommend updating the Titano documentation, at least for CMA and warn NOT to upgrade the NINA-FW.

Thank you so much. If you need anything from me let me know as well.
Cheers,
Anglerfish

[anglerfish27]

Well it was going so well. No crash all day long ...until..now. New error. Unfortunately It was just connected to USB power not my computer so I don't have any output from thonny. What you see is all I have. I checked the SD card immediately. It was clean, empty except for the image cache file it creates. Based in the msg maybe the image itself had something to do with it processing incorrectly? I dont know. I guess I will connect Thonny to it and let it run over night so if an error is thrown I might get more info. See attached message.

[anglerfish27]

I just noticed the URL is in there at the top!. Might help for a repro..

[anglerfish27]

https://openaccess-cdn.clevelandart.org/2003.105/2003.105_web.jpg

Image loads fine on my browser.