Skip to content

kubelet v1.8.4 ppc64le musl-c gets exception #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
runlevel5 opened this issue Jun 26, 2020 · 5 comments
Closed

kubelet v1.8.4 ppc64le musl-c gets exception #13

runlevel5 opened this issue Jun 26, 2020 · 5 comments

Comments

@runlevel5
Copy link

runlevel5 commented Jun 26, 2020 •
edited
Loading

Steps to reproduce:

  1. Compile kubelet v1.18.4 with golang 1.14.4
  2. Run kubelet

Expected results

No error

Actual results

$ sudo kubelet
I0626 12:31:19.173178   25297 server.go:417] Version: v1.18.4
I0626 12:31:19.173388   25297 plugins.go:100] No cloud provider specified.
W0626 12:31:19.173407   25297 server.go:560] standalone mode, no API client
W0626 12:31:19.178562   25297 info.go:51] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
W0626 12:31:19.178914   25297 server.go:474] No api server defined - no events will be sent to API server.
I0626 12:31:19.178977   25297 server.go:647] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
I0626 12:31:19.179495   25297 container_manager_linux.go:266] container manager verified user specified cgroup-root exists: []
fatal error: missing deferreturn

runtime stack:
runtime.throw(0x1394064e, 0x13)
	/usr/lib/go/src/runtime/panic.go:1116 +0x5c
runtime.addOneOpenDeferFrame.func1.1(0x7fff8e01f7e8, 0x0, 0x164145a0)
	/usr/lib/go/src/runtime/panic.go:753 +0x258
runtime.gentraceback(0x1004fc44, 0xc000bbf310, 0x0, 0xc000000180, 0x0, 0x0, 0x7fffffff, 0x7fff8e01f8b8, 0x0, 0x0, ...)
	/usr/lib/go/src/runtime/traceback.go:334 +0xea0
runtime.addOneOpenDeferFrame.func1()
	/usr/lib/go/src/runtime/panic.go:721 +0x8c
runtime.systemstack(0x0)
	/usr/lib/go/src/runtime/asm_ppc64x.s:269 +0x94
runtime.mstart()
	/usr/lib/go/src/runtime/proc.go:1041

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/lib/go/src/runtime/asm_ppc64x.s:216 +0x10 fp=0xc000bbf1f0 sp=0xc000bbf1d0 pc=0x10068f20
runtime.addOneOpenDeferFrame(0xc000000180, 0x1004fc44, 0xc000bbf310)
	/usr/lib/go/src/runtime/panic.go:720 +0x7c fp=0xc000bbf240 sp=0xc000bbf1f0 pc=0x1003649c
panic(0x130d14a0, 0x163c1f20)
	/usr/lib/go/src/runtime/panic.go:929 +0xdc fp=0xc000bbf310 sp=0xc000bbf240 pc=0x10036adc
runtime.panicmem(...)
	/usr/lib/go/src/runtime/panic.go:212
runtime.sigpanic()
	/usr/lib/go/src/runtime/signal_unix.go:695 +0x3f4 fp=0xc000bbf350 sp=0xc000bbf310 pc=0x1004fc44
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/resource.(*Quantity).String(0x0, 0x138b5520, 0x0)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/resource/quantity.go:601 +0x28 fp=0xc000bbf410 sp=0xc000bbf370 pc=0x104cf068
fmt.(*pp).handleMethods(0xc000ac16c0, 0xc000000076, 0x101)
	/usr/lib/go/src/fmt/print.go:630 +0x28c fp=0xc000bbf6b0 sp=0xc000bbf410 pc=0x100e8bfc
fmt.(*pp).printValue(0xc000ac16c0, 0x138b5520, 0xc000c88060, 0x196, 0x76, 0x5)
	/usr/lib/go/src/fmt/print.go:727 +0x211c fp=0xc000bbf8a0 sp=0xc000bbf6b0 pc=0x100eba7c
fmt.(*pp).printValue(0xc000ac16c0, 0x132a6780, 0xc000c88060, 0x199, 0x76, 0x4)
	/usr/lib/go/src/fmt/print.go:810 +0x170c fp=0xc000bbfa90 sp=0xc000bbf8a0 pc=0x100eb06c
fmt.(*pp).printValue(0xc000ac16c0, 0x13573520, 0xc000c88040, 0x199, 0x76, 0x3)
	/usr/lib/go/src/fmt/print.go:810 +0x170c fp=0xc000bbfc80 sp=0xc000bbfa90 pc=0x100eb06c
fmt.(*pp).printValue(0xc000ac16c0, 0x12d40ec0, 0xc0002e32c0, 0x97, 0x76, 0x2)
	/usr/lib/go/src/fmt/print.go:869 +0x3dc fp=0xc000bbfe70 sp=0xc000bbfc80 pc=0x100e9d3c
fmt.(*pp).printValue(0xc000ac16c0, 0x136725a0, 0xc0002e3280, 0x99, 0x76, 0x1)
	/usr/lib/go/src/fmt/print.go:810 +0x170c fp=0xc000bc0060 sp=0xc000bbfe70 pc=0x100eb06c
fmt.(*pp).printValue(0xc000ac16c0, 0x13860f20, 0xc0002e3200, 0x99, 0x76, 0x0)
	/usr/lib/go/src/fmt/print.go:810 +0x170c fp=0xc000bc0250 sp=0xc000bc0060 pc=0x100eb06c
fmt.(*pp).printArg(0xc000ac16c0, 0x13860f20, 0xc0002e3200, 0x76)
	/usr/lib/go/src/fmt/print.go:716 +0x2a8 fp=0xc000bc02f8 sp=0xc000bc0250 pc=0x100e91f8
fmt.(*pp).doPrintf(0xc000ac16c0, 0x13a0cb08, 0x3b, 0xc000bc06b8, 0x1, 0x1)
	/usr/lib/go/src/fmt/print.go:1030 +0x140 fp=0xc000bc0408 sp=0xc000bc02f8 pc=0x100ec5a0
fmt.Fprintf(0x141063e0, 0xc000388310, 0x13a0cb08, 0x3b, 0xc000bc06b8, 0x1, 0x1, 0x1, 0x1391c4be, 0x13860f20)
	/usr/lib/go/src/fmt/print.go:204 +0x58 fp=0xc000bc0480 sp=0xc000bc0408 pc=0x100e5cc8
k8s.io/kubernetes/vendor/k8s.io/klog.(*loggingT).printf(0x164cd8e0, 0xc000000000, 0x13a0cb08, 0x3b, 0xc000bc06b8, 0x1, 0x1)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:703 +0x98 fp=0xc000bc0510 sp=0xc000bc0480 pc=0x1030e868
k8s.io/kubernetes/vendor/k8s.io/klog.Infof(...)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:1201
k8s.io/kubernetes/pkg/kubelet/cm.NewContainerManager(0x141ae680, 0xc0002a1660, 0x141d6ee0, 0xc000a90b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cm/container_manager_linux.go:271 +0x560 fp=0xc000bc0a20 sp=0xc000bc0510 pc=0x1151f730
k8s.io/kubernetes/cmd/kubelet/app.run(0xc0008ba000, 0xc000501500, 0x7fff8def1440, 0xc00007db00, 0xc0000d4a80, 0x1, 0x1)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:715 +0xd60 fp=0xc000bc19d8 sp=0xc000bc0a20 pc=0x12a62ac0
k8s.io/kubernetes/cmd/kubelet/app.Run(0xc0008ba000, 0xc000501500, 0x7fff8def1440, 0xc00007db00, 0xc0000d4a80, 0x0, 0x1016d30c)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:421 +0xfc fp=0xc000bc1b38 sp=0xc000bc19d8 pc=0x12a615cc
k8s.io/kubernetes/cmd/kubelet/app.NewKubeletCommand.func1(0xc0000fb180, 0xc0001121a0, 0x0, 0x0)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:273 +0x51c fp=0xc000bc1d10 sp=0xc000bc1b38 pc=0x12a675bc
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc0000fb180, 0xc0001121a0, 0x0, 0x0, 0xc0000fb180, 0xc0001121a0)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:830 +0x208 fp=0xc000bc1df8 sp=0xc000bc1d10 pc=0x128b7b38
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0000fb180, 0x161c17ede727a2d3, 0x164cd2a0, 0x1003989c)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914 +0x258 fp=0xc000bc1ee0 sp=0xc000bc1df8 pc=0x128b8528
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
main.main()
	_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:41 +0xdc fp=0xc000bc1f50 sp=0xc000bc1ee0 pc=0x12a6860c
runtime.main()
	/usr/lib/go/src/runtime/proc.go:203 +0x214 fp=0xc000bc1fc0 sp=0xc000bc1f50 pc=0x10039914
runtime.goexit()
	/usr/lib/go/src/runtime/asm_ppc64x.s:884 +0x4 fp=0xc000bc1fc0 sp=0xc000bc1fc0 pc=0x1006b644

goroutine 19 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x164cd8e0)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:1010 +0x78
created by k8s.io/kubernetes/vendor/k8s.io/klog.init.0
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:411 +0xe0

goroutine 88 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x14115940, 0xc000a90b70, 0xc00030df20)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:288 +0x98
created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:286 +0x68

goroutine 82 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.SetupSignalHandler.func1(0xc0000d4a80)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/signal.go:38 +0x38
created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.SetupSignalHandler
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/signal.go:37 +0xd8

goroutine 45 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/lib/go/src/runtime/time.go:188 +0xc4
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1.(*metricsRecorder).run(0xc000371f20)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1/metrics_recorder.go:87 +0x40
created by k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1.newMetricsRecorder
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1/metrics_recorder.go:59 +0xf4

goroutine 32 [syscall]:
os/signal.signal_recv(0x0)
	/usr/lib/go/src/runtime/sigqueue.go:147 +0xf8
os/signal.loop()
	/usr/lib/go/src/os/signal/signal_unix.go:23 +0x24
created by os/signal.Notify.func1
	/usr/lib/go/src/os/signal/signal.go:127 +0x4c

goroutine 57 [select]:
k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.(*worker).start(0xc000698820)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/go.opencensus.io/stats/view/worker.go:154 +0xd8
created by k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.init.0
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/go.opencensus.io/stats/view/worker.go:32 +0x64

goroutine 80 [select]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x13ab54e0, 0x14105a20, 0xc00044e150, 0x1, 0xc0001020c0)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x120
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x13ab54e0, 0x12a05f200, 0x0, 0xc000126901, 0xc0001020c0)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x90
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Forever(0x13ab54e0, 0x12a05f200)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x50
created by k8s.io/kubernetes/vendor/k8s.io/component-base/logs.InitLogs
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/logs/logs.go:58 +0x88

goroutine 87 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0007324c0)
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch/mux.go:207 +0x58
created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.NewBroadcaster
	/home/vagrant/aports/testing/kubernetes/src/kubernetes-1.18.4/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch/mux.go:75 +0xc4

Extra info

os: alpine 3.12 (ppc64le)
go 1.14.4
libc: musl-c
@runlevel5
Copy link
Author

Any guidance on how to provide more debug information is greatly appreciated

@runlevel5
Copy link
Author

UPDATE: This issue only occur in go v1.14. The upstream go has a fix https://go-review.googlesource.com/c/go/+/234105/

@runlevel5
Copy link
Author

So while waiting for the v.1.14.5, users who are impacted by this issue could compile kubelet with the latest minor release v1.13

@runlevel5
Copy link
Author

runlevel5 commented Jul 16, 2020 •
edited
Loading

kubelet (compiled with v1.14.5) still run into the same issue.

It turns out this is not the same as https://go-review.googlesource.com/c/go/+/241087/, I am going to re-open this issue instead.

@runlevel5 runlevel5 reopened this Jul 16, 2020
@runlevel5 runlevel5 mentioned this issue Jul 16, 2020
Closed
@runlevel5
Copy link
Author

It is resolved with golang 1.14.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@runlevel5