proposal: cmd/go: add go test -long

[danderson]

Abstract

Add go test -long as a gate for (very) long tests.

Problem statement

I write MetalLB, a set of Go programs that deploy into Kubernetes and do marginally useful things within. Currently, it's tested entirely by short and fast unit tests, but I would very much like to add integration tests that run the code against Kubernetes.

This involves spinning up a local Kubernetes cluster, which takes several minutes, and building and pushing several Docker images, taking another minute or so. This means these tests take 2-5 minutes to even start running, and probably another minute or two to finish.

With the current Go tool UX, the default behavior is the opposite of desirable for rapid iteration, and I often find myself accidentally spinning up this giant coal plant of a test during iterative development. This is an irritating series of little paper cuts.

Empirically, people are solving this on their own. In my code I check for a "RUN_SLOW_TESTS" env var, and use that to t.Skip the long tests. It works well enough, but everyone implementing their own hacks for this means the developer experience is fragmented, and it's easier to miss the fact that long tests exist at all, because the "run absolutely everything" switch is different for everyone.

Proposal

In a perfect world, remove go test -short and replace it with go test -long, flipping the default to be "you must choose to run the very slow things. Given that this breaking change is likely not palatable, I propose adding go test -long in addition to the existing go test -short.

In effect, this creates 3 bands of tests: short, medium, and long, with "short and medium" being the default go test runs. -short and -long act as lowering or raising a bar of "how much I'm willing to wait": -short only allows the short band to run, whereas -long allows short, medium and long to run.

Again I'm not in love with the 3 bands and would prefer 2 with a flipped default. Given that there would be three, I'd propose the following rules of thumb for what tests go where:

• go test -short should be as fast as the compiler, fast enough that I can run compile+test interactively as part of an IDE.
• go test should be fast enough that the developer's attention doesn't wander off to Twitter. Call it less than 10s total.
• go test -long is for persnickety CI pipelines and coffee breaks. Minutes, tens of minute, hours if you really must (though I sincerely hope nobody does that last one).

cc @bradfitz

[bitfield]

If this were implemented, running simply go test would miss some tests, and you'd have to know that long tests were present (and about the -long flag!) in order to run them. The safer option is to always run all tests unless you knowingly opt out of the long ones, per the status quo.

[mvdan]

I have mixed feelings about this proposal. I too wish that long/integration tests were more standard across Go projects. On the other hand, I'm not sure that go test -long would bring much value. At least from my personal experience, long tests tend to require extra secrets or resources to run, and users don't tend to spare minutes to run a project's tests.

It seems to me like go test -long would mainly be a minor convenience for a project's own active contributors, but not really for users running go test in general.

[danderson]

@bitfield The problem is that, empirically, tests are already getting skipped. I have an if os.Getenv("SKIP_SLOW_TESTS_" != "" { t.Skip(...) }, and I'm told that others have implemented similar empirical methods for excluding long tests from go test.

So, the question is whether it's better to keep a dozen different empirical methods, or have a single way to find and run tests that won't run by default.

@mvdan It's true that large tests tend to require more infrastructure. In my case, it requires a bunch of extra installed programs (docker, kind, ...) to build the test harness. So it comes back to: is there value in standardizing the discovery of such tests, even though running them may take more than just go test -long ?

This proposal was filed out of frustration at setting up yet another janky harness for E2E tests, but I'm not deeply married to solving the problem, so this discussion is very useful!

[iand]

The usual way of segregating tests is via build tags. Add a tag to mark long tests and then run them with go test -tags long

[mvdan]

is there value in standardizing the discovery of such tests

I think so. Perhaps we can simply standardize an existing way to do it, such as @iand's build tag idea, or your environment variable idea. This would still make most projects alike, without having to overcomplicate the go test setup and flags for everyone.

If we go with a build tag, my only suggestion would be to make it more specific than long, such as longtest.

[bcmills]

CC @julieqiu @dmitshur

[bcmills]

It seems clearly useful to be able to skip non-hermetic tests (such as those that require unavailable tools), although for those you can generally use exec.LookPath or similar probes to skip the test if a dependency is missing.

With a bit of work (and assuming that we fix #33976 soon), you can even make the skip contingent on your own module not being the main module (so that developers on your project always run all of your tests, and the tests fail a needed tool is missing). See golang.org/x/tools/testenv for an example.

So I think we should be careful not to conflate “long” with “non-hermetic”.

[bcmills]

go test -long is for persnickety CI pipelines and coffee breaks. Minutes, tens of minute, hours if you really must (though I sincerely hope nobody does that last one).

If you have three granularities, why not more? In the Go standard library, we have at least four:

1. go test -short is about as fast as the compiler for most packages, but for more involved binaries (such as cmd/go) it degrades to “not so long as to be a distraction”. If the short tests pass, the toolchain is likely good enough to bootstrap itself but may still contain subtle bugs and regressions.

2. go test is fast-ish for most packages, but slow for large ones. It runs enough tests to catch most regressions in ordinary configurations.

3. all.bash runs go test -short for most packages, plus several additional tests for integration with other tools (such as cgo) and configurations (such as varying values of GOMAXPROCS).

4. GO_BUILDER_NAME guards platform-specific or very expensive tests. Some of those only run on specific builders in the Go project's CI.

That's not to say that every project should follow this structure. However, it does suggest that -long is not particularly universal or general.

[bcmills]

That said: the user has an explicit control to tell you how long they expect tests to run: the -timeout flag, which should always be accurate as of Go 1.13 (see #28147).

So one option you could pursue would be to tailor the set of Skip calls based on flag.Lookup("test.timeout").Value: if it indicates a very high timeout, run the very long tests, and if it indicates a very short timeout, then skip them.

[mvdan]

I really like @bcmills's last idea. After all, -short and -long are all relative, and different projects will have different points of reference.

I'd love it if this was high-level, like testing.Short. For example, imagine if one could query the order of magnitude that the test is able to take to run for. Imagine an API like func (*testing.T) Magnitude() time.Duration, which could return powers of ten (100 * time.Millisecond, time.Second, 10 * time.Second, etc). This could be the order of magnitude right below -test.timeout, for example.

The reason I bring up orders of magnitude is because I can often tell if tests will take milliseconds, seconds, or minutes to run. But it's hard to say if they will take 1s or 2s, because the CPU/disk/network/etc of different machines could easily vary by 5x. Using a cutoff that directly used -test.timeout would probably result in failures on slow machines.

[ianlancetaylor]

See also #30595 (a rejected proposal) and #31310.

[bcmills]

But it's hard to say if they will take 1s or 2s, because the CPU/disk/network/etc of different machines could easily vary by 5x. Using a cutoff that directly used -test.timeout would probably result in failures on slow machines.

It shouldn't depend much on the machine: you should only be running the test at all if the timeout is 1–2 orders of magnitude longer than how long you expect the test to run, for precisely that reason.

My touchstone for this is user behavior: if a user ran your program, how long would they wait before hitting ⌃C or canceling a page-load? That's how long the test should be willing to wait, too.

There is a bit of a question as to how you divvy up -test.timeout among all of the tests in the file, but arguably the easy heuristic is to always base the decision on how much time is left rather than how much time you started with.

[bcmills]

arguably the easy heuristic is to always base the decision on how much time is left rather than how much time you started with.

On that note, see #28135, which I guess is on me to implement. 😅