Launching JVM alpine profiler pod failed

[italux]

I'm trying to use the flame with a JVM-based image running into an EKS cluster with Kubernetes version 1.15, but every time I try to run the profiling I got the error bellow

kubectl flame POD -f /tmp/flamegraph.svg -n NAMESPACE -l java --alpine CONTAINER
Verifying target pod ... ✔
Launching profiler ... ❌
pod failed

Looking into the failed POD logs I got the error bellow

{"type":"progress","data":{"time":"2020-12-11T01:43:28.02366121Z","stage":"started"}}
{"type":"error","data":{"reason":"exit status 255"}}

Also, I've attached the failed POD describe

Name:           kubectl-flame-6a3fb104-aaf4-4f86-92a0-2c53c7f6601c-tqpft
Namespace:      NAMESPACE
Priority:       0
Node:           ip-172-20-124-211.ec2.internal/172.20.124.211
Start Time:     Thu, 10 Dec 2020 22:43:26 -0300
Labels:         controller-uid=9fcbed51-3f1c-42fb-af18-118e53b1a3cf
                job-name=kubectl-flame-6a3fb104-aaf4-4f86-92a0-2c53c7f6601c
                kubectl-flame/id=6a3fb104-aaf4-4f86-92a0-2c53c7f6601c
Annotations:    kubernetes.io/psp: eks.privileged
                sidecar.istio.io/inject: false
Status:         Failed
IP:             172.20.126.116
IPs:            <none>
Controlled By:  Job/kubectl-flame-6a3fb104-aaf4-4f86-92a0-2c53c7f6601c
Containers:
  kubectl-flame:
    Container ID:  docker://ee90601c752f38721ecd9c4d3ce8fd0bf5aaecadfbfa385c524d9514b35762e2
    Image:         verizondigital/kubectl-flame:v0.1.5-jvm
    Image ID:      docker-pullable://verizondigital/kubectl-flame@sha256:463c8ef5075c42644818c7646bc6e9371905a4253248a8f3be1c7acfa146c343
    Port:          <none>
    Host Port:     <none>
    Command:
      /app/agent
    Args:
      6a3fb104-aaf4-4f86-92a0-2c53c7f6601c
      7f317a90-b1d9-4772-ae1e-ec429df6cdf1
      CONTAINER
      docker://6ab1c5da9dca6267bf24f82691d3464825dff639ec6b7d83d25af77716ff6e39
      1m0s
      java
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 10 Dec 2020 22:43:28 -0300
      Finished:     Thu, 10 Dec 2020 22:43:28 -0300
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/lib/docker from target-filesystem (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p67s5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  target-filesystem:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker
    HostPathType:
  default-token-p67s5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p67s5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason   Age    From     Message
  ----    ------   ----   ----     -------
  Normal  Pulling  4m52s  kubelet  Pulling image "verizondigital/kubectl-flame:v0.1.5-jvm"
  Normal  Pulled   4m52s  kubelet  Successfully pulled image "verizondigital/kubectl-flame:v0.1.5-jvm"
  Normal  Created  4m52s  kubelet  Created container kubectl-flame
  Normal  Started  4m51s  kubelet  Started container kubectl-flame

[edeNFed]

Hi @italux
Can you please tell me which JVM version you using in the target container?
Thanks

[italux]

Hi @italux
Can you please tell me which JVM version you using in the target container?
Thanks

Hey @edeNFed, thanks for the answer, below the JVM version:

/ # java -version
openjdk version "1.8.0_111-internal"
OpenJDK Runtime Environment (build 1.8.0_111-internal-alpine-r0-b14)
OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)

[dontmint]

I meet the same problem for Java, Python profiler runs just fine, Golang not tested
My Java version is :

java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)```

[italux]

I realized that this happen because even the agent and target container doesn't have the libstdc++, see:

# /tmp/async-profiler/profiler.sh -d 60 -f /tmp/flamegraph.svg -e wall 1
Failed to inject profiler into 1
        ldd (0x7f1ebca35000)
Error loading shared library libstdc++.so.6: No such file or directory (needed by /tmp/async-profiler/build/libasyncProfiler.so)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7f1ebc1e2000)
        libc.musl-x86_64.so.1 => ldd (0x7f1ebca35000)

When the target container has the libstdc++, everything "works fine":

Verifying target pod ... ✔
Launching profiler ... ✔
Profiling ... Got invalid event: invalid character '/' looking for beginning of value
✔
FlameGraph saved to: /tmp/farinha-flamegraph.svg 🔥

The #47 is an attempt to solve the issue, but I still struggle with the libstdc++ need on the target container as well, any creative ideas on how to solve this? @edeNFed @dontmint

[saltbo]

Is there any progress on this issue?

[italux]

Officially alpine is not supported anymore by the official OpenJDK distribution as we can see below:

The OpenJDK port for Alpine is not in a supported release by OpenJDK, since it is not in the mainline code base. It is only available as early access builds of OpenJDK Project Portola. See also this comment. So this image follows what is available from the OpenJDK project's maintainers.

With that in mind, I'm moving all application to use a openjdk image (non-alpine) and I'm not sure if the effort to fix this issue is really valid, what you think @edeNFed?

[italux]

I've decided to close my proposal, maybe @saltbo would like to propose a fix here

[saltbo]

I suddenly realized that I had another problem. I will going to reopen an issue.

Failed to inject profiler into 147932
	linux-vdso.so.1 (0x00007fff57388000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f33097b2000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3309595000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f330938d000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f330900b000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3308d07000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3308af0000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3308751000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f3309dfb000)

[saltbo]

#57