v9.2.2 thru v9.2.4 slows to a crawl on ESP32-S2 when both ssd1306 OLED display and AHT20 sensor on the Stemma I2C bus at the same time

[Timeline8]

CircuitPython version and board name

v9.2.2 thru v9.2.4
Adafruit QT Py ESP32-S2 WiFi Dev Board with STEMMA QT Product ID: 5325

Code/REPL

import time
import board
import displayio
import i2cdisplaybus
import adafruit_displayio_ssd1306  # OLED
import adafruit_ahtx0  # WeAct T&H Sensor
import adafruit_sht4x  # Adafruit sensor

time.sleep(2)  # give me time to open serial window after power

# set up I2C with OLED display & AHT20 sensor
i2c = board.STEMMA_I2C()
displayio.release_displays()
display_bus = i2cdisplaybus.I2CDisplayBus(i2c, device_address=0x3C)
display = adafruit_displayio_ssd1306.SSD1306(display_bus, width=128, height=64)
ali_sensor = adafruit_ahtx0.AHTx0(i2c)  # WeAct AHT20-F Sensor
ada_sensor = adafruit_sht4x.SHT4x(i2c)  # Adafruit 5776 Sensor

for i in range(5):
    print("Loop ", i)
    time.sleep(0.3)

displayio.release_displays()

Behavior

Full description on my post to the Adafruit forums here: https://forums.adafruit.com/viewtopic.php?p=1044638

When both the OLED and the AHT20 are both initialized my code then crawls along very slowly. This does not happen pre-9.2.2.

Description

If I only initialize one or the other (OLED or AHT20) it works fine. I then added a SHT41 sensor to see if it is simply any two devices on the bus. But only the OLED+AHT20 combination slowed down. Any one device individually worked normal speed. OLED+SHT41 worked normal speed. And AHT20+SHT41 worked normal speed. Note in my code, I didn't even need to read from either. I simply printed text to the REPL to see the slow down. Note: for the display I commented out all three display setup lines. I didn't try individual lines.

This does not happen pre-9.2.2 so when I loaded in 9.2.1 it worked normal speed. Switching to 9.2.2 I get the slow down. Switching back to 9.2.1 normal speed again. When I first discovered this yesterday with a larger program it ran normal on 9.1.3, 9.2.0 & 9.2.1 but 9.2.2, 9.2.3, & 9.2.4 all has the slow down issue.

Appears to be limited to the S2. I then tried this with a Qt Py S3 and had no issues with any combination running CP v9.2.4

Additional information

settings.toml file was renamed so there was no connection to Wifi for this testing.

Testing set up was Qt Py S2 Stemma to Adafruit Stemma 5-port passive hub (Product ID: 5625) with Adafruit SHT41 sensor and WeAct Studio AHT20 sensor both plugged into the bug. OLED plugged into protoboard with Stemma to "Dupont" wires to also connector to hub.

[dhalbert]

This may be totally unrelated, but in espressif/esp-idf#14603 I found an ESP32-S2 problem with the old I2C driver. We switched to the new driver, but I wonder if there is a regression of some kind in ESP-IDF or how we use it.

[dhalbert]

I tested this with a QT Py ESP32-S2, an AHT20, and a 128x32 OLED SSD1306 display, https://www.adafruit.com/product/4440. I did not attach an SHT41. The I2C devices were daisy-chained.

I could not get it to act up. I also tried a Feather ESP32-S2 with the AHT20 and the display above, and also two FeatherWing SSD1306 displays, 128x64 and 128x32.

I tested both with a code.py that ran immediately and with a test.py that I invoked via the REPL.

Do you have another SSD1306 OLED display you could try? EDIT: also, it looks like the AHT20 is not the Adafruit board (I used an Adafruit board). Does it have pull-ups, and if so, what are their values?

EDIT #2: I added an SHT40. Still fine.

[Timeline8]

I actually ran it (unintentionally) on two different displays but both the same type other than one is white and the other yellow. First one is on my aquarium monitoring setup and was the one I discovered the issue with and the one in the video (link on the Adafruit forum post). The following day when I put together the same setup on my desk and then used the other OLED. It did the same thing.

However, worth noting I was using the same Qt Py S2 for all testing. The first time playing with it I had performed an

import storage
storage.erase_filesystem()

on the board as I had some issue with file storage space not clearing when deleting files and then the next day continued to use the same board.

Perhaps I should grab the S2 again and this time do a full erase with the Adafruit ESPTool online tool to really erase the entire board?

[Timeline8]

OK just performed a full erase, update bootloader (18.2 to 20.1), load CP v9.2.4 UF2, copied the code above from here into MU, and used Circup to install the ssd1306, ahtx0, and sht4x libraries. I didn't put anything else on the board. It still behaves the same.

A new thing I noticed, which I think I forgot to try before, is if I initialize all three, it also runs normally. Comment out the SHT41 initilization line 17 and the OLED+AHT20 combo starts crawling again.

The main difference between my setup and yours are that my OLED and AHT20 are not Adafruit products (the SHT41 is Adafruit). And I am hooked up in star formation (per the photo above) vs. your daisy chain. The WeAct Studio AHT20 is using 4.7k pull-up resistors on the I2C bus. It is also designed for 3.3V to 5V and has a SOT23 IC onboard which I am assuming is a 3.3V LDO regulator plus supporting caps and another resistor (330 ohms).

And to reiterate, I didn't unplug anything so all pull ups from the AHT & SHT sensors stayed in circuit. And I did not comment out the libraries so all three libraries always loaded for each combination attempted. I will have to check if the OLED has pullups as well. But again, if it does, all three sets were present for all testing.

Let me know if there is anything else I can do or try.

[Timeline8]

Sorry for another follow-up. When I measured the I2C bus total pull up it isn't make sense. Then I realized while the AHT20 has its own pullups, they are not in parallel with the rest of the bus because the onboard LDO between 3.3V input and 3.3V output on the board. So total pull up value for all three devices measured while disconnected from the S2 is a bit over 6.1k which is a reasonable value. Just wanted to mention it for completeness. However since I never unplugged any of them and only changed which ones got initialized that shouldn't make a difference. But also it doesn't explain why v9.2.2 - .4 does this but not pre-9.2.2 versions when using the same hardware.

[dhalbert]

It is true I am doing a daisy-chain but the daisy-chain is really just parallel, like your STEMMA/QT breakout. Electrically it is the same except for the wire lengths. This is the program I am using, really the same as yours. I just commented out the SHT4X() and it didn't help.

If you have another SSD1306 display I would say try that.

I have heard of counterfeit SHT30 boards but not AHT20.

import time
import board
import displayio
import i2cdisplaybus
import adafruit_displayio_ssd1306  # OLED
import adafruit_ahtx0  # WeAct T&H Sensor
import adafruit_sht4x  # Adafruit sensor

time.sleep(2)  # give me time to open serial window after power

# set up I2C with OLED display & AHT20 sensor
i2c = board.STEMMA_I2C()
displayio.release_displays()
display_bus = i2cdisplaybus.I2CDisplayBus(i2c, device_address=0x3C)
display = adafruit_displayio_ssd1306.SSD1306(display_bus, width=128, height=32)
ali_sensor = adafruit_ahtx0.AHTx0(i2c)  # WeAct AHT20-F Sensor
#ada_sensor = adafruit_sht4x.SHT4x(i2c)  # Adafruit 5776 Sensor

for i in range(10):
    print("Loop ", i)
    time.sleep(0.3)

displayio.release_displays()

[Timeline8]

I have heard of counterfeit SHT30 boards but not AHT20.

Well I wouldn't call it a counterfeit. It is a custom board made by WeAct using the same 3rd party's AHT20 sensor. If WeAct had simply taken the Adafruit PCB design and reproduced it verbatim, that would be a counterfeit. Designing their own PCB around the same 3rd party sensor is not.

On the OLED subject, I only have the two OLEDs of the same design. While I love Adafruit products, and try to support you guys by buying as much Adafruit stuff as I can, Adafruit's monochrome OLED product selections are simply overpriced to me. The cheapest (ready to use) monochrome OLED is smaller, low ppi, & more expensive than a larger higher density full color TFT, so I am unlikely to pick up an Adafruit OLED breakout board. I would choose a 1.14" 240x136 TFT all day long from Adafruit over any of the OLEDs.

[dhalbert]

I should have been more precise. I have worked with users who had boards that seemed to have counterfeit SHT30 chips (they acted oddly and/or had different I2C addresses). I have not heard of a counterfeit AHT20 chip.

(Adafruit's boards are open source. The layout can be copied. If the board is marked as Adafruit, then that's counterfeit. But if it's the same design, but does not pretend to be Adafruit, it's just a copy.)

Do you have any other sensor boards that provoke this slow behavior? I am just trying to find something that I can reproduce so I can debug it.

[Timeline8]

Ah gotcha on the sensor chip being the part you meant that is counterfeited. For whatever it is worth, I have that same WeAct sensor on 4 long running setups (all running CircuitPython code) and was using another on a temporary test running a Qt Py with a lipo using an Adafruit Charger BFF running Wippersnapper. I haven't had any issue with them up until the recent Qt Py S2 update to 9.2.4. Two of the other three besides this one are S3 boards and no problems. The remaining one is an Adafruit REV TFT Feather S2 and I haven't attempted that one yet. But that one of course has a built in display. When I have time, I can try to update that one to 9.2.4 first and see how it goes. If it's OK I can try some test code to use the OLED instead of the built in display and see how that goes. Maybe this weekend.

I am trying to remember what else in I2C I might have. I do have the Adafruit EEPROM (Product ID: 5146) I bought on a whim I could hook up. Actually now that I think about it that one might be on this same Aquarium setup being used as an impromptu Stemma hub without using the EEPROM. So I could add that library and initiation line to the code. Not a sensor, but at least another I2C device.

On my next DigiKey order I will have to add an Adafruit AHT20 and see if simply swapping out the AHT20 boards with the exact same code makes a difference. But might be a little while before I am able to put in an order.

[dhalbert]

Could you give a sample URL for the display you bought? I'll see if there's anything unusual about it.

[dhalbert]

Do you have an oscilloscope or logic analyzer so you could check the SCL clock frequency or otherwise compare 9.2.1 with later versions?

[Timeline8]

WeAct Studio has a storefront on everyone's favorite Chinese site ;) and here is the specific sensor listing https://www.aliexpress.us/item/3256806723829530.html

I don't own a scope, nor logic analyzer, of my own, but I am sure I can borrow a scope from work (not sure what we have for logic analyzers). Another "perhaps this weekend" task.

[dhalbert]

How about a link for the display? Thanks.

[Timeline8]

Are you not seeing the link in my last post?
That is the purchase link.

Also see here https://github.com/WeActStudio/WeActStudio.AHTxxModule/tree/master

[dhalbert]

I am asking about the display, the OLED SSD1306 board.

[Timeline8]

Duh! Sorry. Me read gooder. ;)

Not that the colors matter much but the white one I bought here...
https://www.aliexpress.us/item/3256806808213099.html
and the yellow one here...
https://www.aliexpress.us/item/3256806186748120.html

[dhalbert]

I found a display (generic, not Adafruit-branded) just like that in my collection. It works fine. So I think it's something about the AHT20. Given that taking that away with all other combinations fixes the problem, it's causing a stall of some kind apparently.

[Timeline8]

Oddly though it does work when I initialized all three (OLED, AHT, & SHT) which is just weird.

Well if you can't reproduce it, I guess for now we can leave it be until new info comes up from someone else or if I pick up an Adafruit version of the AHT20 I will report back. For now that setup is running 9.2.1 which works and if I ever need to update it further I can switch to an S3 or perhaps just throw a TFT at it. This issue isn't a critical one for me so I can work around it.

Thanks for looking into it.

[dhalbert]

One more thing to try:

import busio
#...
i2c = busio.I2C(board.SCL1, board.SDA1, frequency=400000)
#...

It may work better at higher or lower frequencies. If not at 400000, try 200000, and 50000. 100000 is the default.

Note that you have to use SCL1 and SDA1. SCL and SDA are not connected to the STEMMA/QT port on the QT Py ESP32-S2. They make a separate bus connected to the SCL/SDA pins on the board edge.

[Timeline8]

Thanks Dan, I will give that a try later. Also, I had to make a DigiKey order this morning so I threw in an Adafruit AHT20 breakout board on there, so I will try that first and see how it acts. If it acts normal then that indeed points to the WeShare version & I will then try your last suggestion on the clock speed. On the other hand, if it slows down as well, then we have a mystery why my setup slows and your nearly identical setup does not. Anyway, I will post back when I get new information to share.

[Timeline8]

OK back with more info but not really helpful. I got the Adafruit AHT20 yesterday. I first upgraded my S2 back to 9.2.4 and verified again that

i2c = board.STEMMA_I2C()
displayio.release_displays()
display_bus = i2cdisplaybus.I2CDisplayBus(i2c, device_address=0x3C)
display = adafruit_displayio_ssd1306.SSD1306(display_bus, width=128, height=64)
ali_sensor = adafruit_ahtx0.AHTx0(i2c)  # WeAct AHT20-F T&H Sensor
ada_sensor = adafruit_sht4x.SHT4x(i2c)  # Adafruit 5776 Sensor

works fine with all three initialized, works fine if I comment out the three "display" lines (so two temp sensors initialized), and works fine if I comment out AHT2x (so OLED+SHT4x initialized). And then with the SHT4x commented out so I initialize OLED+AHT2x, it slows to a crawl again. So nothing different than reported above last week.

I then swapped out the WeShare AHT20 and plugged in the Adafruit AHT20. There was no change in behavior for each of the three scenarios above. So it would appear the WeShare part is innocent.

I then added print ("Unplug Oled") and a 2 second pause between the above code and the 5 iteration loop (because the OLED needs to be present when you initialize it and can't run with it unplugged from the start). Verified once more it still crawled if I did nothing but run the modified code (print & sleep statements). I then ran it again and this time pulled the OLED from the breadboard. It ran at full speed again. Verified a couple times.

And just for kicks, since the Adafruit AHT20 has two Stemma connectors, I removed the stemma hub and plugged everything in series; QTPY->AHT20->AHT41->Breadboard with OLED but it made no difference as expected.

Finally after changing the code so only the OLED+SHT20 are initialized and reconfirming the crawl speed. I then went into the bootloader, loaded CP v9.2.1, and when it was done the same code ran normal speed again.

So in summary it would appear the OLED is the catalyst under 9.2.4 however very confusing why it does it when only the AHT2x is initialized but works perfectly fine when all three were initialized. Oh and works fine by itself, which I think I forgot to mention. That is it. I don't expect anyone to do anything further since Dan couldn't get this same setup to do what I am seeing, so just recording all this in the chance that someone else find the same or a similar problem and runs across this issue report while trying to figure out their problem or someone else plays with it and can reproduce it.

[bablokb]

Do you think you could give this logic-analyzer a try: https://github.com/gusmanb/logicanalyzer

It is very simple to use, you basically need a Pico for it (the logic-analyzer is on top, the DUT with the AHT20 below):

You will see something like this in the analysis software:

As you can see, this is with version 4, but they are already at version 6. Maybe this will give some additional insights.

[dhalbert]

@Timeline8 Could you try adding some delays around the initialization of each I2C device? Something like 0.5 seconds should be more than adequate.

[Timeline8]

@bablokb, that is pretty neat. I will have to try it out when I get a chance. Is the project compiled strictly for Pico boards (for their onboard hardware and more specifically their pinouts) or can it be adapted to any RP2040 board?

@dhalbert, I will try adding some delays perhaps later today and see if I come up with any different behavior. And now that I think about it, I can also try change the order in which those devices are initiated to see if that does anything different.

[bablokb]

@bablokb, that is pretty neat. I will have to try it out when I get a chance. Is the project compiled strictly for Pico boards (for their onboard hardware and more specifically their pinouts) or can it be adapted to any RP2040 board?

I think they support Pico, Pico-W and Pico2 plus a Zero-variant, I think from Waveshare. The firmware for the logic-analyzer is not written in Python, so probably portable but not configurable.

[Timeline8]

I think they support Pico, Pico-W and Pico2 plus a Zero-variant, I think from Waveshare. The firmware for the logic-analyzer is not written in Python, so probably portable but not configurable.

Darn. I do have one Pico W board but it is currently in use in my basement. I have a Seeed XIAO RP2040 (bought by accident) not in use so thought maybe that would work. But if not, since I have been meaning to make some changes to that basement Pico setup anyway, might be a good opportunity to switch it over to one of the ESP32-S2/S3 boards I already have and then freeing up the Pico to put together the logic analyzer. Or just crack open the wallet and spend another whopping (/s) $6 to get a second Pico W (or $4 for the non-W).

I will have to read up more on the project. Thanks.

[bablokb]

Or just crack open the wallet and spend another whopping (/s) $6 to get a second Pico W (or $4 for the non-W).

I would really recommend the Pico-W, since the analyzer software can control the device via wlan and this greatly simplifies the mechanical setup. I also use a small LiPo and thus I can just place the whole package next to the DUT and connect it with a few cables. See https://github.com/bablokb/3D-pico-lipo-shim