I2C/Wire/TwoWire can lock up or cease working in some conditions

[Curclamas]

Bug description

Steps to reproduce

Given the following minimum code example:

#include "Wire.h"

void setup() {
  Wire.begin(21,22);
  Serial.begin(115200);
  Serial.println("Start i2c-test"); 
}

void loop() {
  byte error;

  // 0x20 is the address of a pcf8574 GPIO expander
  Serial.println("Try to contact 0x20"); 

  Wire.beginTransmission(0x20);
  error = Wire.endTransmission();

  Serial.print("Error code is:");
  Serial.println(error);

  delay(1000);
}

We've run this on a Nano32, WROOM32 and on a custom board. Run it:

• on a proper (10k) pulled up I2C-bus with no devices on it
• on a bus that has PCF8574 on it
• on a bus that is only a wire with no pullups in place
• on a bus where SCL is pulled up and SDA is pulled down

also try to change the condition of the test during the runtime of the code (e.g. remove a pullup or a cable on a breadboard).

Observed behavior

1. Initially we see error code 0 (no error, i.e. device ACKed) or 2 (NACK on address). After removing a wire or pull up we see error code 3 (NACK on data (⚠️ we have not even sent data here ⚠️)). When we then put in the missing component again the outputted error code will stay 3 until we reset the ESP32.

2. If we pull down SDA the code execution will stop / there will be no more stuff printed to the console.

3. Observing what happens on an oscilloscope we see in the first case (removed pullup) that after the faulty code (3) has occured the first time the I2C bus stays dark (high) on SCL and SDA.
   In the second case (SDA pulled down) we observed a steady oscillation at 100kHz on SCL until we reset the device and remove the pulldown on SDA. Removing the pulldown alone does not work.

4. We have observed an error condition as in (1.) on a known good board after several days of polling a value from a PCF8754 every 20ms. It looks like if the request fails once it will fail over and over again until we reboot.

Expected behavior

• When we have a faulty connection or device on the I2C bus the Wire commands MUST return a meaningful error code and MUST NOT freeze the ESP32/code execution (maybe timeout or throw an exception if inevitable?).
• Also when we have removed the faulty hardware condition and retry the command it MUST return a 0 (if we have an i2c device) or 2 (if no device is present) error code. I.e. it MUST be resilient against faulty conditions without a reboot.

[me-no-dev]

Excellent bug report!

[Curclamas]

I've done a little bit more research on that using a DSO.
While doing that I've noticed quite some strange things.

First Screenshot: Spurious signals

Still given that example code above and just having a proper terminated bus with no other devices the first thing we see is:

What we see here: (Yellow is SCL and blue is SDA)

• A spurious pulse on SDA and then SCL
• Then we have a I2C transmission that looks like READ to addr 0x00 (what is that, general call addr? why?) with a NACK on the end
• Then starts what looks like the expected: first addr: 0b100000 (0x20 in 7 bit) then 0b0 (read) then 0b1 (NACK) and then the 0b1 from the STOP condition (P)

This bothers me a little since I don't understand where the first pulse and the first strange package comes from. I know that spurious signals can fuck up the state machines of some I2C-sensors but I'm certainly no I2C-Expert. Also on a production board I am routinely running an I2C-scan on boot, sometimes it shows devices which are clearly not there, maybe that could have something to do with that?

Here are the two packages up close:

Second Screen: Lockup if SDA gets pulled low

In the following screenshot we see what happens if we pull SDA low (spike comes from bouncing) between two Packages that are 1000ms apart. You see the first package (just as a spike because of the zoom). Then you see the second message. Afterwards SDA gets pulled low (AFAIK that can happen in I2C if the bus is "busy").
We can see that after the bus has been pulled low no more spikes can be seen on SCL. That applies to SCL and SDA when that pulldown is removed and the bus goes back to normal operation. The system is locked then

Third screen: What happens with wire.begin() in the loop() ?

I was curios to know what happens if you put wire.begin() in the loop() . Because sometimes the system would not lock up but rather report error 3 all the time.
What happens then is: we always get those strange two packages from Screen one (the single pulse only occurs on the first package after boot) everytime.

If we have a faulty condition the device will lock up or - if it does not lock up - will still respond error 3 and not attempt do do any more activity on the I2C bus:

[me-no-dev]

Yes I know about that first pulse. It comes from the hardware. I have tried many things to make it not happen, but all unsuccessful. I think I put a bit of delay there to help the device differentiate it from actual data, but that is as much as I can do about it. As for the slave pulling SDA low... that usually means that the slave has stuff to do and will release the data line once done. Used in many devices.

[me-no-dev]

Teo (Espressif's boss) just gave me an idea :) I'll give it a go and report. If you do not hear from me in the next 3 days, please bug me to do it. I am not in my country currently and will return tomorrow, so there is a chance that I will forget.

[CarlosGS]

I've had similar problems with i2c. In my case the device locks up when SDA keeps pulled low.

This also happened with ESP8266, maybe the recovery code could be reused: esp8266/Arduino#1025

[me-no-dev]

Looking at my code, I have already come up with the same idea... that is that strange transmission at init. A transmission that aims to reset the bus.

[lonerzzz]

I have been fighting with this problem myself for a almost a week now. What I have observed is the following:

• when the error occurs, the bus stays in a busy state, regardless of whether the SDA and SCL lines are recovered, preventing subsequent messages from being sent. The busy state is from the bus_busy parameter of the status_reg in this file: https://github.com/espressif/esp-idf/blob/master/components/soc/esp32/include/soc/i2c_struct.h
• single byte reads following after single byte writes most often lead to the problem
• conversely a single byte write followed by a five byte read results in the problem happening less frequently for the same overall number of messages
• separating the read from the preceding write by a delay of 3 microseconds is enough to drop the frequency of the error considerably (this has me wondering if we have a variable that is maybe not volatile so not being read directly all the time?)
• when the I2C gets into the stuck state, there is often an inability for WiFi to be established after a reboot so wondering if flash or a peripheral is sharing the first I2C bus and contributing to problems)

[me-no-dev]

what if we give it timeout and then manually set bus_busy to 0?

[lonerzzz]

Either or both of those are valid options for testing. I am willing to test anything you want to try. I am getting familiar with the esp32-hal-* files but haven't ventured into the idf code very deep just yet. Let me know how you wish to proceed.

[me-no-dev]

@lonerzzz I went through the code and I do not see me checking after write if bus_busy is 0. It is done only in case of error. I have the feeling that I did this on purpose but my memory of the case is not that good. Maybe we need to make sure that the bus is not busy at the end of write, wait a bit and set it to 0 if it fails (then return some error I guess).

[lonerzzz]

@me-no-dev I have been trying a number of things, but I am not able to reset the state such that the flag i2c->dev->status_reg.bus_busy gets set back to 0. I have also observed that in a number of situations, this flag is already set before i2cInitFix is even called just after starting to boot. I added code in i2cInitFix to check bus_busy because the i2cInitFix code was sitting in the tight loop at the end of the method after a reasonable percentage of reboots and preventing system startup. I am using the ESP32 as an I2C master and nothing is using the bus in my application prior to the Wire.begin() call.

[muktillc]

I am not being able to generate the I2C clock on the pin. Don't know why the simple clock is also not present. I have the following code.

Hardware:

Board: ESP32 Dev Module
Core Installation/update date: 11/jul/2017
IDE name: Arduino IDE
Flash Frequency: 40Mhz
Upload Speed: 115200

Description:

I am not able to generate the I2C clock. I have an ST microelectronics accelerator that I am using which has an I2C interface. So just wanted to see if I2C is working.

I have seen many people running into the same problem but could not find a solution for this issue. No errors generated when compiling the code.

Please find my code below.

Sketch:

//Change the code below by your sketch

#include "Wire.h"

void setup() {
Wire.begin(21,22);
Serial.begin(115200);
Wire.setClock(400000); // choose 400 kHz I2C rate
Serial.println("Start i2c-test");
}

void loop() {
byte error;

// 0x20 is the address of a pcf8574 GPIO expander
Serial.println("Try to contact 0x20");
uint8_t data = 0;
// Wire.beginTransmission(0x3A);
// error = Wire.endTransmission();
Wire.beginTransmission(0x3B); // Initialize the Tx buffer
Wire.write(0x0F); // Put WHO_AM_I address in Tx buffer
Wire.endTransmission(false); // Send the Tx buffer, but send a restart to keep connection alive
Wire.requestFrom(0x3B, 1); // Read two bytes from slave PROM address
while (Wire.available()) {
data = Wire.read(); } // Put read results in the Rx buffer
Serial.print("Error code is:");
Serial.println(data);

delay(1000);
}