Infix 25.02 broken on NanoPi R2S

[troglobit]

User reports issue upgrading R2S from 25.01 to 25.02:

I just updated my farm of r2s from 25.01.0 to 25.02.0 and getting Corrupt startup-config
I built a new image from 25.02 and dd the sdcard, same result

In 25.02 we made two syntax changes to existing YANG models that stepped the version of confd to 1.5, this can be seen in .cfg files in the "infix-meta:meta" node. To handle syntax changes we have introduced jq scripts that migrate the startup-config of deployed machines when they first start up with a new confd version. So in this case, upgrading from 25.01 to 25.02, two scripts run:

• src/confd/share/migrate/1.5/10-change-key-types.sh
• src/confd/share/migrate/1.5/11-rename-etherlike.sh

The latter is for Qemu systems, so possibly the issue is with converting the SSH host key type.

[troglobit]

Reproduced.

Upgrading one of my R2S shows only:

[ OK ] Bootstrapping YANG datastore
[ OK ] Starting Configuration daemon
[FAIL] Loading startup-config
[ OK ] Starting Configuration daemon
[ OK ] Verifying self-signed https certificate
[ OK ] Starting Status daemon
[ OK ] Stopping Status daemon
[WARN] Killing Status daemon
[ OK ] 
[ OK ] Update DNS configuration
[ OK ] Starting Status daemon

Infix -- a Network Operating System v25.02.0 (ttyS2)

ERROR: Corrupt startup-config, system has reverted to default login credentials
failed-00-00-00 login:

Further study of logs show:

Apr  6 10:39:09 infix migrate[946]: /cfg/startup-config.cfg: backup created: /cfg/backup/startup-config-0.0.cfg
Apr  6 10:39:09 infix migrate[946]: /cfg/startup-config.cfg: migrating from version 0.0
Apr  6 10:39:09 infix migrate[946]: Applying migrations for version 1.0 ...
Apr  6 10:39:09 infix migrate[946]: /cfg/startup-config.cfg: calling /usr/share/confd/migrate/1.0/10-infix-shell-type.sh ...
Apr  6 10:39:09 infix migrate[946]: Applying migrations for version 1.1 ...
Apr  6 10:39:09 infix migrate[946]: /cfg/startup-config.cfg: calling /usr/share/confd/migrate/1.1/10-infix-routing-type.sh ...
Apr  6 10:39:09 infix migrate[946]: Applying migrations for version 1.2 ...
Apr  6 10:39:09 infix migrate[946]: /cfg/startup-config.cfg: calling /usr/share/confd/migrate/1.2/10-ietf-interfaces-phys-address.sh ...
Apr  6 10:39:09 infix migrate[946]: Applying migrations for version 1.3 ...
Apr  6 10:39:09 infix migrate[946]: /cfg/startup-config.cfg: calling /usr/share/confd/migrate/1.3/10-ssh-server.sh ...
Apr  6 10:39:09 infix migrate[946]: Applying migrations for version 1.4 ...
Apr  6 10:39:09 infix migrate[946]: /cfg/startup-config.cfg: calling /usr/share/confd/migrate/1.4/10-dhcp-client-option-name.sh ...
Apr  6 10:39:09 infix migrate[946]: /cfg/startup-config.cfg: calling /usr/share/confd/migrate/1.4/20-dhcp-client-option-remap.sh ...
Apr  6 10:39:09 infix migrate[946]: Applying migrations for version 1.5 ...
Apr  6 10:39:09 infix migrate[946]: /cfg/startup-config.cfg: calling /usr/share/confd/migrate/1.5/10-change-key-types.sh ...
Apr  6 10:39:09 infix migrate[946]: /cfg/startup-config.cfg: calling /usr/share/confd/migrate/1.5/11-rename-etherlike.sh ...
Apr  6 10:39:09 infix migrate[946]: /cfg/startup-config.cfg: configuration updated to version 1.5.
Apr  6 10:39:10 infix ixinit[272]: [ OK ]  /usr/libexec/infix/init.d/30-cfg-migrate
Apr  6 10:39:10 infix ixinit[272]: Calling /usr/libexec/infix/init.d/99-done ...
Apr  6 10:39:10 infix ixinit[272]: [ OK ]  /usr/libexec/infix/init.d/99-done
Apr  6 10:39:10 infix finit[1]: Starting bootstrap[1059]
Apr  6 10:39:11 infix bootstrap[1252]: /etc/sysrepo/factory-default.json: found and fixed old syntax!                           <---- This part looks good!
Apr  6 10:39:13 infix finit[1]: Starting confd[1257]
Apr  6 10:39:14 infix finit[1]: Starting startup[1266]
Apr  6 10:39:14 infix load[1266]: Loading startup-config ...
Apr  6 10:39:15 infix startup[1266]: [WRN] Obsolete schema node "meta" instantiated in data.
Apr  6 10:39:15 infix startup[1266]: [WRN] Obsolete schema node "version" instantiated in data.
Apr  6 10:39:15 infix startup[1266]: [WRN] Obsolete schema node "meta" instantiated in data.
Apr  6 10:39:15 infix startup[1266]: [WRN] Obsolete schema node "version" instantiated in data.
Apr  6 10:39:15 infix confd[1257]: Username "admin" is valid
Apr  6 10:39:16 infix dagger[1257]: Aborting: /run/net/0/action/init/lan/10-ethtool-flow-control.sh failed with exitcode 78     <---- Here's the likely root cause
Apr  6 10:39:17 infix dagger[1257]: Abandoned generation 0
Apr  6 10:39:17 infix confd[1257]: Failed to apply interface configuration
Apr  6 10:39:17 infix confd[1257]: Oups, error detected in SR_EV_DONE
Apr  6 10:39:17 infix confd[1257]: Processing subscription events failed (Internal error).
Apr  6 10:39:17 infix finit[1]: Stopping startup[1266], sending SIGTERM ...
Apr  6 10:39:17 infix finit[1]: Service confd[1257] died, restarting in 2000 msec (1/10)
Apr  6 10:39:17 infix finit[1]: Starting confd[1390]
Apr  6 10:39:17 infix finit[1]: Starting mkcert[1394]
Apr  6 10:39:17 infix finit[1]: Stopping mkcert[1394], sending SIGTERM ...
Apr  6 10:39:17 infix finit[1]: Starting statd[1395]
Apr  6 10:39:17 infix finit[1]: Stopping statd[1395], sending SIGTERM ...
Apr  6 10:39:17 infix statd[1395]: Status daemon starting
...

... and after that the system tries to start up on failure-config, as intended.

[troglobit]

Further study shows:

root@failed-00-00-00:/tmp/fail/0/action/init/lan$ cat 10-ethtool-flow-control.sh.log 
ethtool: parameter '--json' can be used only with netlink
autoneg unmodified, ignoring
rx unmodified, ignoring
tx unmodified, ignoring
no pause parameters changed, aborting
root@failed-00-00-00:/tmp/fail/0/action/init/lan$ uname -a
Linux failed-00-00-00 6.10.3 #1 SMP PREEMPT Sun Apr  6 06:57:28 CEST 2025 aarch64 GNU/Linux

[wkz]

I have some vague memories of:

1. You disabling ethtool netlink support in the kernel config for R2S, to work around some confd issue
2. Me running into the same issue with iMX8MP-EVK
3. Someone fixing the original issue

If (and that is a big if) I remember all of the above correctly, might this issue be fixed if you just enable ethtool netlink support again?

[troglobit]

I have some vague memories of:

1. You disabling ethtool netlink support in the kernel config for R2S, to work around some confd issue
2. Me running into the same issue with iMX8MP-EVK

Aha didn't know this, or I forgot!

3. Someone fixing the original issue

Interesting, that world be good. I'm attempting to reduce the differences now.

If (and that is a big if) I remember all of the above correctly, might this issue be fixed if you just enable ethtool netlink support again?

Very possible, I'm rebuilding the kernel with exactly that change now. We'll see the result in the morning. 😌

[troglobit]

@wkz yup, you were right, of course! Found a couple of other nasty problems as well when triaging this issue, fixed on the same branch.