read_namespaced_pod_log() gets stuck after some time

[EvgeniiTitov]

Hi!

I have an Azure k8s cluster where I run workflows with Kubeflow. One of the steps in my pipeline manually creates a pod (kubectl apply ...) and then starts reading logs generated by the container running in the created pod and printing them to the stdout so that they appear in the Kubeflow dashboard. The way I do it is as follows:

def get_logs(self):
    response = self.v1.read_namespaced_pod_log(
        name=self.launcher_pod,
        namespace=self.train_job_namespace,
        follow=True,
        _preload_content=False,
        pretty="true"
    )
    response_gen = response.stream(2 ** 6)
    for line in response_gen:
        try:
            decoded_line = line.decode("utf-8")
        except UnicodeDecodeError:
            print(line)
            continue
        print(decoded_line, end="", flush=True)
    print("INFO:: Logs reading thread has finished")

What I expect to happen - all logs that appear inside the container running in the pod to be successfully extracted and printed to the stdout of the corresponding Kubeflow step (which in turn runs in another pod inside the same cluster).
What happens - I get logs for some time but then they get stuck. If I SSH into the container running in the pod whose logs I want to read I can see the logs but I don't get them using the read_namespaced_pod_log() method running in a different pod.

Anything else we need to know?:
I've tried many different variations of the read_namespaced_pod_log's and stream's parameters but nothing seems to help - it works for a while (the time that passes before it stops printing logs seems to vary as well) and then it gets stuck.

Environment:

• Kubernetes version (kubectl version): 1.20.7
• OS (e.g., MacOS 10.13.6): Alpine Linux v3.14 (3.14.2)
• Python version (python --version) 3.9.6
• Python client version (pip list | grep kubernetes) 17.17.0

Thank you in advance!

Regards,
Evgenii

[roycaihw]

/assign

[roycaihw]

You need to use the watch client to stream the log. Please try this example: https://github.com/kubernetes-client/python-base/pull/93/files#diff-caaa4c6a05b0eef75a61b3c5942aeff261ce4344e51b934efd57227a91586874R83-R91

We should have an example for read_namespaced_pod_log under https://github.com/kubernetes-client/python/tree/master/examples

/kind documentation

[EvgeniiTitov]

Hi @roycaihw,

Thank you for your reply.

Yes, I did try to use the Watch() object to stream the logs at the beginning but I had the same issues - logs get stuck after some time.

Regards,
Evgenii

[roycaihw]

Could you share the code you use to stream the logs with Watch()? And how long do you usually see the stream works before it gets stuck?

[EvgeniiTitov]

Sure @roycaihw , this is the approach that uses Watch()

def get_logs(self):
    w = watch.Watch()

    for line in w.stream(self.v1.read_namespaced_pod_log,
                      name=self.launcher_pod,
                      namespace=self.train_job_namespace,
                      timestamps=True):

        print(line, end="\n")
        if "INFO:: Training completed." in line:
            break

As you can see we rely on reading logs from the pod (self.launcher_pod) in which we're trying our models to signal when training has completed (we are aware it is not the best approach and working on it). If logs never arrive (get stuck), we never see the "INFO:: Training completed." message, as a result our pipeline step hangs indefinitely.

Funny enough, it happens at different times, sometimes I see more logs, sometimes less. Once in a while, it even works as expected. But it is very inconsistent, 99% of the time we just get stuck somewhere while trying to get and print logs received from the launcher_pod.

[naor2013]

@roycaihw any news about this? We also have this issue.

[brendandburns]

You probably need to watch repeatedly. The network connection to stream the logs may become interrupted, and I don't think that the Watch code handles that.

[satish-nubolab]

Changing the value for the variable follow to False resolved the issue for me.

def get_logs(core_v1):
response = core_v1.read_namespaced_pod_log(
name='mypod',
namespace='default',
follow=False,
_preload_content=False,
pretty="true"
)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten