buildg from nerdctl-full crashing go runtime

[apostasie]

Description

61.03 runtime: lfstack.push invalid packing: node=0xece2e7092600 cnt=0x1 packed=0xece2e70926000001 -> node=0xffffece2e7092600
61.03 fatal error: lfstack.push
61.03
61.03 runtime stack:
61.03 runtime.throw({0x119db8a?, 0xc00063f588?})
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/panic.go:992 +0x71
61.03 runtime.(*lfstack).push(0x2?, 0xc00003a020?)
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/lfstack.go:30 +0x125
61.03 runtime.(*spanSetBlockAlloc).free(...)
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/mspanset.go:292
61.03 runtime.(*spanSet).reset(0x1ad4d78)
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/mspanset.go:265 +0x70
61.03 runtime.finishsweep_m()
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/mgcsweep.go:260 +0x9c
61.03 runtime.gcStart.func1()
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/mgc.go:664 +0x17
61.03 runtime.systemstack()
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/asm_amd64.s:469 +0x49
61.03
61.03 goroutine 1 [running, locked to thread]:
61.03 runtime.systemstack_switch()
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/asm_amd64.s:436 fp=0xc00063f490 sp=0xc00063f488 pc=0x462940
61.03 runtime.gcStart({0xece31665b5b8?, 0x5?, 0x1665b5b8?})
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/mgc.go:663 +0x449 fp=0xc00063f510 sp=0xc00063f490 pc=0x419b49
61.03 runtime.mallocgc(0x4, 0x0, 0x0)
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/malloc.go:1205 +0x85c fp=0xc00063f588 sp=0xc00063f510 pc=0x40d47c
61.03 github.com/docker/docker/libnetwork/ipamutils.copyIP(...)
61.03 	/home/runner/go/pkg/mod/github.com/docker/docker@v20.10.3-0.20220414164044-61404de7df1a+incompatible/libnetwork/ipamutils/utils.go:125
61.03 github.com/docker/docker/libnetwork/ipamutils.splitNetwork(0x18, 0xc000360570)
61.03 	/home/runner/go/pkg/mod/github.com/docker/docker@v20.10.3-0.20220414164044-61404de7df1a+incompatible/libnetwork/ipamutils/utils.go:117 +0x1e6 fp=0xc00063f650 sp=0xc00063f588 pc=0xd8ba46
61.03 github.com/docker/docker/libnetwork/ipamutils.splitNetworks({0x1a82f00, 0x1, 0xd0?})
61.03 	/home/runner/go/pkg/mod/github.com/docker/docker@v20.10.3-0.20220414164044-61404de7df1a+incompatible/libnetwork/ipamutils/utils.go:104 +0x13a fp=0xc00063f708 sp=0xc00063f650 pc=0xd8b61a
61.03 github.com/docker/docker/libnetwork/ipamutils.init.0()
61.03 	/home/runner/go/pkg/mod/github.com/docker/docker@v20.10.3-0.20220414164044-61404de7df1a+incompatible/libnetwork/ipamutils/utils.go:36 +0x32 fp=0xc00063f730 sp=0xc00063f708 pc=0xd8b412
61.03 runtime.doInit(0x1a34120)
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/proc.go:6222 +0x126 fp=0xc00063f860 sp=0xc00063f730 pc=0x443e86
61.03 runtime.doInit(0x1a3e320)
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/proc.go:6199 +0x71 fp=0xc00063f990 sp=0xc00063f860 pc=0x443dd1
61.03 runtime.doInit(0x1a3e640)
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/proc.go:6199 +0x71 fp=0xc00063fac0 sp=0xc00063f990 pc=0x443dd1
61.03 runtime.doInit(0x1a47000)
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/proc.go:6199 +0x71 fp=0xc00063fbf0 sp=0xc00063fac0 pc=0x443dd1
61.03 runtime.doInit(0x1a46360)
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/proc.go:6199 +0x71 fp=0xc00063fd20 sp=0xc00063fbf0 pc=0x443dd1
61.03 runtime.doInit(0x1a485c0)
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/proc.go:6199 +0x71 fp=0xc00063fe50 sp=0xc00063fd20 pc=0x443dd1
61.03 runtime.doInit(0x1a45ae0)
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/proc.go:6199 +0x71 fp=0xc00063ff80 sp=0xc00063fe50 pc=0x443dd1
61.03 runtime.main()
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/proc.go:233 +0x1d3 fp=0xc00063ffe0 sp=0xc00063ff80 pc=0x436f33
61.03 runtime.goexit()
61.03 	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00063ffe8 sp=0xc00063ffe0 pc=0x464a21

Command ran was buildg --version (on an arm64 machine, running an amd64 version of it with qemu, inside a container 😰)

The stack trace is a bit wild.
I do not like to see libnetwork over there - but then my (total spitballing) read is that it is a bug in golang GC.
Maybe it has to do with the interaction with qemu as well.
Either way this is likely not worth the time spent debugging it, given it is an old golang version.

@ktock ever seen this?

@AkihiroSuda @ktock should we switch from downloading buildg binary releases to compiling it instead (with a more recent golang version)?

Steps to reproduce the issue

Describe the results you received and expected

na

What version of nerdctl are you using?

main

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

Notes:

• quite likely a qemu (and/or golang) issue
• possibly this is (currently open) discussed here: runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64 golang/go#69255

[apostasie]

So, outside a container, with Rosetta, core dump.

Not completely sure this is the same issue - the Rosetta angle is apparently a kernel regression wrt vdso (discussed at lima: lima-vm/lima#2891 ).

Possibly
torvalds/linux@48f6430

[apostasie]

Issues are unrelated:

• rosetta works fine with a pre-6.11 kernel without the vdso patch
• with qemu, the binary is still crashing

[apostasie]

The QEMU angle of the issue is possibly related to page size.
Apple M1 runs on a 16kb page size, while lima runs a 4kb page size kernel.

My understanding is that qemu does not like it when the host and guest page sizes differ.

The next step would be to run a custom kernel on lima to validate the above ^.

I am giving up on this though (just dropping Ubuntu and QEMU). Rosetta works fine enough with Debian 12 (with pre 6.11 kernel).

Virtualization in 2025 is off to a very, very wrong start...

Going to close this ticket since this is not buildg fault, and there is nothing we can do about it here.