testing: want a way to timeout an individual test

[nigeltao]

The testing package provides a timeout flag, "if positive, sets an aggregate time limit for all tests", which defaults to zero (no timeout).

In https://go-review.googlesource.com/#/c/8998/ I wanted a timeout for an individual TestFooBar test. It wasn't too hard to implement myself, but it should be part of the standard testing library.

[minux]

I think what we really want is checking forward progress, not a absolute time limit (and previous experience showed that no matter how we set the timeout value, there are always slower machines that will use up the time.) I think it's especially true for timeouts for a single test. I don't have a proposal though.

[cespare]

I occasionally write tests for constructs which, if they are buggy, could deadlock. I prefer to have my own timeout because then (a) the user can quickly recognize that the test has failed after only, say, 1 second instead of waiting for a very long timeout (default of 10 minutes), and (b) the message can clearly describe the failure.

For an example of such a test, I think that TestWaitGroup in the stdlib would hang for the default timeout on certain kinds of bugs. (Maybe small timeouts wouldn't be desirable for standard library tests given @minux's point about needing to run on arbitrarily slow machines, but hopefully it's an illustrative example.)

It would be nice to do t.Timeout(time.Second) rather than explicitly making a goroutine in such cases.

[rsc]

In CL 8998 your solution depends on moving the work that might never finish into its own goroutine, so that the goroutine running the test can still call t.Fatalf and exit. A plain t.Timeout API is not sufficient here because it would not capture the goroutine issue. One could imagine a more complex API like t.MustRunInTime(time.Duration, func()), but that could just as easily be provided by a helper library. It doesn't need to be in package testing. It's hard for me to see why this must be in package testing.

(In contrast, only package testing knows enough about the aggregate test execution to implement the timeout flag.)

If you want to continue with this, I would suggest resubmitting as a proposal. A design doc will probably be needed.