x/crypto/ssh: client.NewSession can hang indefinitely

[mborsz]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.9.3

Does this issue reproduce with the latest release?

I'm not able to verify this.

What operating system and processor architecture are you using (go env)?

linux, amd64

What did you do?

In kubernetes e2e we are using ssh to fetch logs from kubernetes nodes.
In kubernetes/kubernetes#66609 we see that it quite frequently hangs for ~90 minutes in client.NewSession call (the stacktrace is there).

Relevant code is available here: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/log_size_monitoring.go#L245

What did you expect to see?

Attempt to create NewSession should finish with error if node doesn't respond to ssh connection.

What did you see instead?

Attempt to create NewSession hung for ~90 minutes.

Relevant stacktraces are available in:

• gce scalability tests stuck on gathering log sizes kubernetes/kubernetes#66609
• https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable2/414/build-log.txt (starts with SIGABRT: abort)

[agnivade]

/cc @hanwen

[hanwen]

What is the problem here? Your stack trace suggests it's waiting for the remote end to acknowledge the SSH session. If you want timeouts, you should implement them separately.

Arguably, the SSH package should support contexts to do this neatly, but I think it might be an invasive change, API wise.

[mborsz]

Thanks for the response!

What is the problem here? Your stack trace suggests it's waiting for the remote end to acknowledge the SSH session. If you want timeouts, you should implement them separately.
The problem is that there is no way (AFAIK) to prevent openChannel from blocking for hours in case the remote end never acks the SSH session.

I would like to see some timeout mechanism there. Could you hint how can I implement timeout there?

Arguably, the SSH package should support contexts to do this neatly, but I think it might be an invasive change, API wise.

[ymmt2005]

@mborsz
We could avoid the problem by making a raw TCP net.Conn first and wrapping it
with ssh.NewClientConn. You can set any deadline to the raw connection before
calling SSH methods.

https://github.com/cybozu-go/cke/pull/81/files is the fix.

[hanwen]

The SSH state machine doesn't support timeouts on a channel level. The only thing you can do is tear down the entire SSH connection if there is an error.

[EthanHemo]

You can see the problem in much smaller scale than k8s.

1. Run command nc -lv 8090 locally
2. Execute the following code:

sshConfig := &ssh.ClientConfig{
	HostKeyCallback: ssh.InsecureIgnoreHostKey(),
	Timeout:         10 * time.Second,
}
_, err := ssh.Dial("tcp", "<local_ip>:8090", sshConfig)
if err != nil {
	panic(err)
}

This will run indefinitely since the logic [email protected]/ssh/transport.go:332:

_, err := io.ReadFull(r, buf[:])
if err != nil {
	return nil, err
}

is waiting for an input to read, but it won't get any since it open locally.

While this issue not likely to occur, it can cause DOS attack.