cmd/go: frequent test timeouts on dragonfly-amd64 builder

[bcmills]

2021-03-24T03:16:14-4357f71/dragonfly-amd64
2021-03-23T23:09:33-87a3ac5/dragonfly-amd64
2021-03-23T01:21:24-d25476e/dragonfly-amd64
2021-03-19T17:31:59-4829031/dragonfly-amd64
2021-03-19T16:11:47-a937729/dragonfly-amd64
2021-03-18T21:27:21-d3ab6b5/dragonfly-amd64
2021-03-18T21:22:04-bdbba22/dragonfly-amd64
2021-03-18T17:14:39-f47fab9/dragonfly-amd64
2021-03-18T16:51:27-eaa1dde/dragonfly-amd64
2021-03-18T16:51:24-af4388a/dragonfly-amd64
2021-03-18T04:17:00-c2d6251/dragonfly-amd64

The timeouts don't seem to occur in any one particular test case, and no individual test case seems to be running for an inordinately long time. I think this builder is just slow, but I'm filing this issue in case there is a specific performance problem in cmd/go or the test itself that we can address.

I wonder if something about the dragonfly kernel or the builder configuration in particular makes the I/O syscalls used by cmd/go slower than on other platforms.

CC @tuxillo @jayconrod @matloob

[tuxillo]

Probably it is the setup. Each builder is running in a VM but the virtual disks are stored in hard disks, not SSDs. Also, we have to periodically cleanup the filesystem more often than in a standard setup, in which the cleanup occurs once a day. If they are causing too much trouble, I could probably move the VMs to another server. Let me know!

[dmitshur]

Based on that information about the builder, it sounds like it's expected it to need more time to complete tests successfully, so increasing the test timeout (#45216) has a good chance of resolving this issue.

[gopherbot]

Change https://golang.org/cl/315329 mentions this issue: dashboard: increase GO_TEST_TIMEOUT_SCALE for dragonfly builders

[tuxillo]

This one can be closed?

[bcmills]

Unfortunately no.

2021-06-14T07:12:21-3249b64/dragonfly-amd64
2021-06-13T23:53:43-14305bf/dragonfly-amd64
2021-06-12T00:12:55-9d46ee5/dragonfly-amd64
2021-06-11T21:43:54-e552a6d/dragonfly-amd64
2021-06-11T16:09:15-e2dc6dd/dragonfly-amd64
2021-06-09T22:49:08-27f8372/dragonfly-amd64
2021-06-09T02:18:24-07ca28d/dragonfly-amd64
2021-06-08T22:35:56-bcecae2/dragonfly-amd64
2021-06-08T20:19:02-689f4c7/dragonfly-amd64
2021-06-03T03:03:28-1c2154a/dragonfly-amd64
2021-06-02T19:45:53-384c392/dragonfly-amd64
2021-06-02T15:04:26-7271753/dragonfly-amd64
2021-06-01T22:58:04-377464f/dragonfly-amd64
2021-05-29T00:21:43-6123e5f/dragonfly-amd64
2021-05-27T19:51:01-eb0b8a1/dragonfly-amd64
2021-05-25T21:54:09-a3eb095/dragonfly-amd64
2021-05-25T10:46:47-874765c/dragonfly-amd64
2021-05-25T04:04:01-98f8936/dragonfly-amd64
2021-05-24T20:38:17-d5449c0/dragonfly-amd64
2021-05-24T16:41:39-e64a37c/dragonfly-amd64
2021-05-21T23:19:31-1e0c960/dragonfly-amd64
2021-05-20T15:46:45-3063790/dragonfly-amd64
2021-05-20T14:08:36-0886cdd/dragonfly-amd64
2021-05-19T16:08:23-49064d2/dragonfly-amd64
2021-05-19T14:12:35-a0f4b7b/dragonfly-amd64
2021-05-19T14:12:23-f451690/dragonfly-amd64
2021-05-13T17:35:42-09ab05b/dragonfly-amd64
2021-05-11T20:42:14-be4aaae/dragonfly-amd64

[bcmills]

This builder is still running extremely slowly, and I suspect (but haven't confirmed) that a lot of the problem comes down to disk performance.

We may need to increase the timeout scale factor further, but I wonder if there are other issues at play. For example, is it possible that the builder is memory-starved? Maybe we need to artificially lower GOMAXPROCS to avoid hitting swap while running the tests...

[tuxillo]

We're going to try and have make.bash working so that the builders can be run in GCE, see: #23060

That might help us a bit with the builder slowness (hopefully).