x/build: `gomote ssh` to plan9-arm fails with `Failed to get IP out of "pi4g": <nil>`

[bcmills]

~/go/src$ go1.19beta1 version -m $(which gomote)

/usr/local/google/home/bcmills/bin/gomote: devel go1.19-b9c4d94fdb Fri Jun 24 18:54:27 2022 +0000
	path	golang.org/x/build/cmd/gomote
	mod	golang.org/x/build	v0.0.0-20220623213349-3ceb9f4e34a9	h1:UXzH6j5xqqdwolS/8sIboizJIjD1FchifnbxqrO4wro=
	dep	cloud.google.com/go/compute	v1.3.0	h1:mPL/MzDDYHsh5tHRS9mhmhWlcgClCrCa6ApQCU6wnHI=
	dep	github.com/aws/aws-sdk-go	v1.30.15	h1:Sd8QDVzzE8Sl+xNccmdj0HwMrFowv6uVUx9tGsCE1ZE=
	dep	github.com/golang/groupcache	v0.0.0-20200121045136-8c9f03a8e57e	h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
	dep	github.com/golang/protobuf	v1.5.2	h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
	dep	github.com/googleapis/gax-go/v2	v2.1.1	h1:dp3bWCh+PPO1zjRRiCSczJav13sBvG4UhNyVTa1KqdU=
	dep	github.com/jmespath/go-jmespath	v0.4.0	h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
	dep	go.opencensus.io	v0.23.0	h1:gqCw0LfLxScz8irSi8exQc7fyQ0fKQU/qnC/X8+V/1M=
	dep	golang.org/x/net	v0.0.0-20220127200216-cd36cc0744dd	h1:O7DYs+zxREGLKzKoMQrtrEacpb0ZVXA5rIwylE2Xchk=
	dep	golang.org/x/oauth2	v0.0.0-20211104180415-d3ed0bb246c8	h1:RerP+noqYHUQ8CMRcPlC2nvTa4dcBIjegkuWdcUDuqg=
	dep	golang.org/x/sync	v0.0.0-20210220032951-036812b2e83c	h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
	dep	golang.org/x/sys	v0.0.0-20220209214540-3681064d5158	h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c=
	dep	golang.org/x/text	v0.3.7	h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
	dep	golang.org/x/time	v0.0.0-20210723032227-1f47c861a9ac	h1:7zkz7BUtwNFFqcowJ+RIgu2MaV/MapERkDIy+mwPyjs=
	dep	google.golang.org/api	v0.70.0	h1:67zQnAE0T2rB0A3CwLSas0K+SbVzSxP+zTLkQLexeiw=
	dep	google.golang.org/genproto	v0.0.0-20220222213610-43724f9ea8cf	h1:SVYXkUz2yZS9FWb2Gm8ivSlbNQzL2Z/NpPKE3RG2jWk=
	dep	google.golang.org/grpc	v1.44.0	h1:weqSxi/TMs1SqFRMHCtBgXRs8k3X39QIDEZ0pRcttUg=
	dep	google.golang.org/protobuf	v1.27.1	h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
	dep	gopkg.in/inf.v0	v0.9.1	h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
	build	-compiler=gc
	build	CGO_ENABLED=1
	build	CGO_CFLAGS=
	build	CGO_CPPFLAGS=
	build	CGO_CXXFLAGS=
	build	CGO_LDFLAGS=
	build	GOARCH=amd64
	build	GOOS=linux
	build	GOAMD64=v1

What did you do?

~/go/src$ gomote create plan9-arm
# host type "host-plan9-arm-0intro" is not elastic; 2 of 1 machines connected, 2 busy
# canceled a post-submit build on a machine of type "host-plan9-arm-0intro"; it should reconnect and get assigned to you
…
~/go/src$ gomote ssh --i-will-not-break-the-host user-bcmills-plan9-arm-0

What did you expect to see?

A successful SSH connection to a builder for the plan9/arm platform.

What did you see instead?

$ ssh -p 2222 mutable-user-bcmills-plan9-arm-0@farmer.golang.org # auth using https://github.com/bcmills.keys
Enter passphrase for key '/usr/local/google/home/bcmills/.ssh/id_rsa':
# Welcome to the gomote ssh proxy, bcmills.
# Connecting to/starting remote ssh...
#
# `gomote push` and the builders use:
# - workdir: /boot/workdir
# - GOROOT: /boot/workdir/go
# - GOPATH: /boot/workdir/gopath
# - env: GO_BUILDER_NAME=plan9-arm
# Happy debugging.
# Plan9 user/pass: glenda/glenda123
# Failed to get IP out of "pi4g": <nil>
Connection to farmer.golang.org closed.

(attn @golang/plan9; CC @golang/release)

[0intro]

Have you succeeded to connect to the plan9-386 builder?

[bcmills]

I didn't try plan9-386 due to #42117.

[gopherbot]

Change https://go.dev/cl/414595 mentions this issue: internal/coordinator/remote: fix log message in legacyIncomingSSHPostAuth

[0intro]

It's likely you will hit the same issue as the one described in #42117, once the current issue will be fixed.

Please try again after CL 414595, so we could see the proper error message.

[cagedmantis]

CL 414595 has been submitted and the coordinator has been deployed.

[0intro]

Is gomote ssh supposed to work on builders running outside of GCE?
Running gomote ssh on an external builder would require it to be accessible from Internet and knowledge of the authentication credentials.

[bcmills]

Is gomote ssh supposed to work on builders running outside of GCE?

Yes.

Running gomote ssh on an external builder would require it to be accessible from Internet and knowledge of the authentication credentials.

I believe that gomote ssh is supposed to tunnel the SSH connection over the buildlet's existing reverse connection. (@cagedmantis and @golang/release could fill you in with more details.)

[0intro]

Would it be possible to obtain access to farmer.golang.org, so I could run gomote on my side?