[R4 WiFi vs Minima, Timers] using timers causing crash on R4 WiFi

[runger1101001]

I have a confusing issue I'm trying to track down involving the timers on Renesas. This is for a motor control library, and I'm trying to initialise the timers on the MCU in very specific ways.

Renesas support has already been of assistance with some details, and it is now working as expected on the UNO R4 Minima, but not on the R4 WiFi.

UNO R4 WiFi
Arduino IDE 2.2.1
Renesas UNO Board Support 1.0.4

The crash occurs a short time after starting the timers:

00:23:57.475 -> Firmware name: "/private/var/folders/h9/sbdx_175559c9zz0n837lkf80000gn/T/arduino/sketches/9F9D4234ED54425EE8E6FF4F4D7142DC/renesas_pwm_test.ino", compiled on: Sep 18 2023
00:23:57.500 -> Fault on interrupt or bare metal(no OS) environment
00:23:57.500 -> ===== Thread stack information =====
00:23:57.500 ->   addr: 20007d88    data: 00000009
00:23:57.500 ->   addr: 20007d8c    data: 00000000
00:23:57.500 ->   addr: 20007d90    data: 00000000
00:23:57.500 ->   addr: 20007d94    data: 00000000
00:23:57.500 ->   addr: 20007d98    data: 00014ebb
00:23:57.500 ->   addr: 20007d9c    data: 20002688
00:23:57.500 ->   addr: 20007da0    data: 00014f36
00:23:57.532 ->   addr: 20007da4    data: 20000f94
00:23:57.532 ->   addr: 20007da8    data: 00000005
00:23:57.532 ->   addr: 20007dac    data: fffffff1
00:23:57.532 ->   addr: 20007db0    data: 00000000
00:23:57.532 ->   addr: 20007db4    data: 000000c1
00:23:57.532 ->   addr: 20007db8    data: 40006000
00:23:57.532 ->   addr: 20007dbc    data: 00000001
00:23:57.532 ->   addr: 20007dc0    data: 03030003
00:23:57.532 ->   addr: 20007dc4    data: ffffffe9

etc...
which decodes to:

runger@Richards-Mac-Pro-2 7-2017q4 % bin/arm-none-eabi-addr2line -e "/private/var/folders/h9/sbdx_175559c9zz0n837lkf80000gn/T/arduino/sketches/9F9D4234ED54425EE8E6FF4F4D7142DC/renesas_pwm_test.ino".elf -a -f 000000c0 0000b686 0000cf9c 0000db94 0000cf9c 0001294c 0000cf9c 0000dade 0000db1e 0000588a 00006256 00006302 00005b3a 00004174 0000e604 0000e646
0x000000c0
??
??:0
0x0000b686
agt_int_isr
/ssd/products-Eslov/FW/MainPrograms/hardware/arduino-git/renesas/extras/e2studioProjects/Santiago/Debug/../ra/fsp/src/r_agt/r_agt.c:1081
0x0000cf9c
_ZN4UART5writeEh
/Users/runger/Library/Arduino15/packages/arduino/hardware/renesas_uno/1.0.4/cores/arduino/Serial.cpp:114
0x0000db94
_ZN7arduino5Print11printNumberEmh
/Users/runger/Library/Arduino15/packages/arduino/hardware/renesas_uno/1.0.4/cores/arduino/api/Print.cpp:258
0x0000cf9c
_ZN4UART5writeEh
/Users/runger/Library/Arduino15/packages/arduino/hardware/renesas_uno/1.0.4/cores/arduino/Serial.cpp:114
0x0001294c
_sbrk_r
:?
0x0000cf9c
_ZN4UART5writeEh
/Users/runger/Library/Arduino15/packages/arduino/hardware/renesas_uno/1.0.4/cores/arduino/Serial.cpp:114
0x0000dade
_ZN7arduino5Print5writeEPKhj
/Users/runger/Library/Arduino15/packages/arduino/hardware/renesas_uno/1.0.4/cores/arduino/api/Print.cpp:35
0x0000db1e
_ZN7arduino5Print5printEPKc
/Users/runger/Library/Arduino15/packages/arduino/hardware/renesas_uno/1.0.4/cores/arduino/api/Print.cpp:66
0x0000588a
_ZN14SimpleFOCDebug7printlnEPKci
/Users/runger/Documents/Arduino/libraries/Arduino-FOC/src/communication/SimpleFOCDebug.cpp:58
0x00006256
_Z18startTimerChannelsP27RenesasHardwareDriverParamsi
/Users/runger/Documents/Arduino/libraries/Arduino-FOC/src/drivers/hardware_specific/renesas/renesas.cpp:307 (discriminator 2)
0x00006302
_Z14_configure3PWMliii
/Users/runger/Documents/Arduino/libraries/Arduino-FOC/src/drivers/hardware_specific/renesas/renesas.cpp:383
0x00005b3a
_ZN14BLDCDriver3PWM4initEv
/Users/runger/Documents/Arduino/libraries/Arduino-FOC/src/drivers/BLDCDriver3PWM.cpp:62
0x00004174
setup
/Users/runger/Documents/Arduino/renesas_pwm_test/renesas_pwm_test.ino:19
0x0000e604
_Z12arduino_mainv
/Users/runger/Library/Arduino15/packages/arduino/hardware/renesas_uno/1.0.4/cores/arduino/main.cpp:117 (discriminator 1)
0x0000e646
atexit
/Users/runger/Library/Arduino15/packages/arduino/hardware/renesas_uno/1.0.4/cores/arduino/main.cpp:139

It's quite unclear to me what is causing the crash.

The agt_int_asr is strange, because I'm using only GPT timers, not AGT. It does always show up in the stack traces though. Also it seems the stack traces always contain printlns (to Serial).

How can we narrow down the cause? What's the difference between Minima and WiFi that causes the error to happen only on the R4 WiFi?

[facchinm]

Hi @runger1101001 , that's very strange (given that the same code is working on the Minima). We use AGT0 as millis() timer, so it's not uncommon to see its isr being called. Would you mind sharing a minimal reproducible sketch that triggers the issue? Thanks a lot!

[Fashion-Corp]

Ciao @runger1101001 , please share the code which produces this error, then I will be happy to check it out. I am awake the next 1 hour, so if you are chap-chap, I will look at it tonight.

The most recent weeks I have seen a bit scattered messages in various fora here dealing with R4, where interrupt routines of various sorts are crashing - and they have all of them Serial.print() statements in them. So. Just share the code here, and I will indeed check it out.

[runger1101001]

Hey, thank you kindly for the quick response.

To prepare a proper example will take me a little bit.

The quick version is this:

• SimpleFOC library dev branch: https://github.com/simplefoc/Arduino-FOC/tree/dev
• Newest Renesas Uno framework v1.0.4, Arduino IDE 2.2.1
• A sketch such as this:

#include "Arduino.h"
#include "SimpleFOC.h"

BLDCDriver3PWM driver = BLDCDriver3PWM(D3, D4, D12, D8);
BLDCMotor motor = BLDCMotor(7);

void setup() {
  Serial.begin(115200);
  SimpleFOCDebug::enable();
  delay(10000);

  Serial.println("Driver init...");
  driver.voltage_power_supply = 10.0f;
  driver.voltage_limit = 5.0f;
  if (driver.init()==0)
    Serial.println("Driver init failed.");

  motor.linkDriver(&driver);
  motor.voltage_limit = 2.4f;
  motor.torque_controller = TorqueControlType::voltage;
  motor.controller = MotionControlType::velocity_openloop;
  motor.init();
  Serial.println("Init complete.");
}

void loop() {
  //Serial.println("Running...");
  delayMicroseconds(100);
  motor.move(10); // 10 rad/s
}

[Fashion-Corp]

I seem not being able to find the origin of the SIMPLEFOC_DEBUG - I cannot find its definition. Which file is SIMPLEFOC_DEBUG() located in?

[runger1101001]

I can package this in a bit better example over the weekend.

The renesas-specific code can be found here: https://github.com/simplefoc/Arduino-FOC/tree/dev/src/drivers/hardware_specific/renesas

Thanks for the information about the AGT. The timer code in question explicitly excludes AGT timers, so there should be no conflict there.

I wonder if the AGT interrupt is actually the problem... perhaps its just confusing me because I'm trying to use the timers and the AGT interrupt routine is on the stack, but actually they're not linked.

I wonder about the top entry on the stack

0x000000c0
??
??:0

Could this be a problem of jumping to a bad address? I was reading bugs related to memory corruption or missing return statements in functions. Perhaps its more to do with something like that?

[runger1101001]

I seem not being able to find the origin of the SIMPLEFOC_DEBUG - I cannot find its definition. Which file is SIMPLEFOC_DEBUG() located in?

Its here: https://github.com/simplefoc/Arduino-FOC/blob/dev/src/communication/SimpleFOCDebug.h

Its intention is to be able to disable all the debug statements per pre-processor to save memory. We're very tight on memory on some MCUs like the original UNO.

[Fashion-Corp]

It furthermore occurs to me, that if the entire of the debug chain of functions is not 100% re-entrant, then you are at risk of re-calling a function in the debug chain of code, which then messes up call stack and everything, leaving you with a crashed code - not always but often.

I cannot see if the code is reentrant all the way through - it looks OK - but I can only say that if you have non-reentrant calls executed from any interrupt-routine, then you see crashes.

I have just looked into SimpleFOCDebug.cpp - and it appeared fine - but you may have to check out the entire call tree.

I think that you would be doing well in checking that part. I may be wrong though. Welcome to write again.

[Fashion-Corp]

Question: Does it ALWAYS crash? Or does it only crash sometimes, and periodically and unpredictably?
Question: Does it crash from an interrupt routine, or without any interrupt routine being invoked?
Question: Have you tried establishing WHERE it crashes in your code? If so - does it always crash from the same place?
Question: Have you tried isolating interrupt-algorithms (by blocking interrupts at certain parts of the code?
Question: The print happening in the debug section - does it call the Serial. print - or where does the output of the print end?

[Fashion-Corp]

ON INTERRUPTS calling non-reentrant code. Just an example into details of what can go wrong:

Could you try to remove all debug printing and all printing as such, from the interrupted code? (comment it out, and mark the comments by starting the line with // RRRR, so that you do not confuse these comments with other comments when you want to reinstate them)

So, if you have an algorithm which is being called both from the normal loop-code, and ALSO from any interrupt-routine - then do not print from it. I have lately seen lots and lots of people who are calling non-reentrant codes in C++ from more than one point. That always fails. 

The problem is roughly this one here with the above: 

The usual call from the tree stemming from the loop() is executing the function, ie. ffffff() 

Then at some point within the ffffff() routine, some non reentrant variables have been set, for instance a counter of characters to print out is reset. The printing starts, and the counter counts up to for instance 17 of the maybe 25 characters to print. Then the interrupt is being called by ie. the timer function. It stops the original printing, pushes all registers of the processor on the stack, but does not preserve variables. So. At the time when the interrupt is calling ffffff(), the counter has reached to 17. Then the interrupt function starts - and calls ffffff() - the counter is reset to 0 again, and starts counting. The interrupt routine is maybe printing a longer string, say, 40 characters where the one printed from the loop is 25. The algorithm runs to the end, and the last character at position 39 is being printed. The interrupt routine returns, the registers restores, and the original instance is continued. Now, however, the counter have reached 39 (character 40 is on index 39). At that point the character 39 is way beyond the end character at position 24 (the 25th character is at index 24). So now the printing continues until a 0 (zero) is met. That can be way into other code, and now whether this crashes is totally up to coincidences. 
Hope it is useful? 

[Fashion-Corp]

@runger1101001 any news - have you got things to work? Or identified where the issue may be?

[runger1101001]

Thanks a lot for all the feedback! Sorry I'm not quite as fast as you are. :-)

Regarding the re-entrant code:
Actually my Renesas PWM driver is not using any interrupts. While we have some code using interrupts in our sensor drivers, this is not being used in the example. So there should be no interrupts in play from my side.

Of course it is a very valid point you make regarding the reentrant code, and using interrupts can be very tricky, so if possible I structure our drivers to avoid them (by using other facilities of the MCUs).

Could you try to remove all debug printing and all printing as such, from the interrupted code?

I'm in the process of doing that, and will report on my findings.

One thought that occurred to me is that there could be a mismatch between the FSP library and the headers I'm compiling against in the core. It's a bit hard to follow that build process, but it seems to me FSP is compiled from a slightly customised branch of v4.0.0. The current version however is v4.6.0, including many bug fixes, some of which affect the timer code.

So last night I was trying to re-compile the framework with FSP v4.6.0 so see if this helps, but its slow going getting that set up.

[Fashion-Corp]

Timers are interruptible by nature??? Aren't you using timers?

…

On Sat, 23 Sep 2023 12:12 runger1101001, ***@***.***> wrote: Thanks a lot for all the feedback! Sorry I'm not quite as fast as you are. :-) Regarding the re-entrant code: Actually my Renesas PWM driver is not using any interrupts. While we have some code using interrupts in our sensor drivers, this is not being used in the example. So there should be no interrupts in play from my side. Of course it is a very valid point you make regarding the reentrant code, and using interrupts can be very tricky, so if possible I structure our drivers to avoid them (by using other facilities of the MCUs). Could you try to remove all debug printing and all printing as such, from the interrupted code? I'm in the process of doing that, and will report on my findings. One thought that occurred to me is that there could be a mismatch between the FSP library and the headers I'm compiling against in the core. It's a bit hard to follow that build process, but it seems to me FSP is compiled from a slightly customised branch of v4.0.0. The current version however is v4.6.0, including many bug fixes, some of which affect the timer code. So last night I was trying to re-compile the framework with FSP v4.6.0 so see if this helps, but its slow going getting that set up. — Reply to this email directly, view it on GitHub <#139 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUI6JVTMGLFEWQZ2FOTG3YDX32RV3ANCNFSM6AAAAAA464EQXQ> . You are receiving this because you commented.Message ID: ***@***.***>

[Fashion-Corp]

What is it you are building with fwm and such stuff?

…

On Sat, 23 Sep 2023 12:12 runger1101001, ***@***.***> wrote: Thanks a lot for all the feedback! Sorry I'm not quite as fast as you are. :-) Regarding the re-entrant code: Actually my Renesas PWM driver is not using any interrupts. While we have some code using interrupts in our sensor drivers, this is not being used in the example. So there should be no interrupts in play from my side. Of course it is a very valid point you make regarding the reentrant code, and using interrupts can be very tricky, so if possible I structure our drivers to avoid them (by using other facilities of the MCUs). Could you try to remove all debug printing and all printing as such, from the interrupted code? I'm in the process of doing that, and will report on my findings. One thought that occurred to me is that there could be a mismatch between the FSP library and the headers I'm compiling against in the core. It's a bit hard to follow that build process, but it seems to me FSP is compiled from a slightly customised branch of v4.0.0. The current version however is v4.6.0, including many bug fixes, some of which affect the timer code. So last night I was trying to re-compile the framework with FSP v4.6.0 so see if this helps, but its slow going getting that set up. — Reply to this email directly, view it on GitHub <#139 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUI6JVTMGLFEWQZ2FOTG3YDX32RV3ANCNFSM6AAAAAA464EQXQ> . You are receiving this because you commented.Message ID: ***@***.***>

[runger1101001]

Question: Does it ALWAYS crash? Or does it only crash sometimes, and periodically and unpredictably?

Always, but not exactly at the same time. Sometimes a few more characters are output by the serial, sometimes less. When I don't change the program, it always seems to crash in the same way. Changing the program can move the moment the crash occurs by a little.

Question: Does it crash from an interrupt routine, or without any interrupt routine being invoked?

No interrupts involved from my side. Only indirectly, if caused by use of Serial, or delayMicroseconds()

Question: Have you tried establishing WHERE it crashes in your code? If so - does it always crash from the same place?

No, it does not. It's very strange in that way.

Question: Have you tried isolating interrupt-algorithms (by blocking interrupts at certain parts of the code?

No, but I will try it.

Question: The print happening in the debug section - does it call the Serial. print - or where does the output of the print end?

Yes, it's an abstraction using the Stream class. But it gets initialised using the standard Serial object.
The SIMPLEFOC_DEBUG is a macro that wraps println so that it can be compiled away, and so it automatically applies the FlashStringHelper. I'm disabling the debug output to see if this is somehow the cause and will report back.

[runger1101001]

So the answer is when I remove all the debug output, and also comment out the delayMicroseconds(), the crash still occurs in both cases.

The simplified test program is now:

#include "Arduino.h"
#include "SimpleFOC.h"

BLDCDriver3PWM driver = BLDCDriver3PWM(D3, D4, D12, D8);

void setup() {
  Serial.begin(115200);
  delay(10000);

  driver.voltage_power_supply = 10.0f;
  driver.voltage_limit = 10.0f;
  if (driver.init()==0)
    Serial.println("Driver init failed.");

  driver.enable();
  delay(1);
  driver.setPwm(2.5f,5.0f,7.5f);
}

void loop() {
  //Serial.println("Running...");
  //delayMicroseconds(100);
}