Skip to content

Watch hang occasionally #884

Closed
Closed
@zhiweiv

Description

@zhiweiv

After a few fixes, the watch api works fine basically, however watch hang still occur occasionally.

The underlying connection was dropped silently even though there is tcp keep alive, then the watch hang at WaitForSocketEvents because timeout is set to Infinite.

In my observation, the api server always close watch connection after 30m~60m, I am thinking set watch timeout to a reasonable value instead of Infinite to fix the hang issue, for example 2 hours, since api server has already closed the connection before this time range in normal condition.

if (watch == true)
{
cts.CancelAfter(Timeout.InfiniteTimeSpan);
}

kubernetes/kubernetes#67491 (comment)
#533 (comment)
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
--min-request-timeout int     Default: 1800

An optional field indicating the minimum number of seconds a handler must keep a request open before timing it out. Currently only honored by the watch request handler, which picks a randomized value above this number as the connection timeout, to spread out load.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions