testing: add M.Cleanup

[matttproud]

This is a feature request/proposal for testing.M. I was doing a Readability Review today of user code. The code for one reason or another was using a custom main entrypoint. I noticed how bare the support in testing.M is for users to report and handle setup/teardown failures as well as conduct cleanups. The setup, execution, and teardown of the custom test main needed to do both of these tasks. It dawned on me that there no advice is given in the documentation about where setup error messages should go and how to correctly abort (e.g., what exit value).

I would like to propose that for similar reasons of testing.T having (*testing.T).Cleanup and (*testing.T).Fatal, testing.M should have the same.

Between the two of them, the cleanup is less important. But how to report errors and setup failures looks very relevant.

For instance, if a test main fails, should the user call log.Fatal or one of its variants, hand emit to stderr or stdout? I have seen the whole zoo of behavior before. Is any non-zero exit value OK or only some? I get that custom test mains are an advanced feature and should be seldom used, but it seems potentially like we are not providing the users an adequately documented journey. Maybe this is intentional (e.g., Go tests to support custom test harness and executor requirement contracts)? It does deserve some Schrift at the very least.

[ianlancetaylor]

It seems to me that testing.M.Cleanup can be implemented simply using defer. This isn't true for testing.T.Cleanup, because of parallel subtests, which can continue to run after the test function completes. So I don't see a strong argument for adding testing.M.Cleanup.

[matttproud]

I agree with that, but I came to suspect a Cleanup facility may be needed if a formal Fatal and exit facility is added to testing.M. One of the triggers for my concern and filing this issue was seeing log.Fatal calls combined with load-bearing defer blocks that wouldn’t run (defer block to reconcile external state that the test mutated in its main entry point and tried to revert). It seems either better documentation or API could ameliorate this since keeping the nuance and semantics in mind as tribal or institutional knowledge simply isn’t sustainable. It also felt wrong to maintain in-house documentation on this as well. I trust your folks judgment. :-)

[rsc]

It sounds like we don't need to worry about an M.Cleanup until/unless we have M.Fatal etc.
And I don't know why we would have M.Fatal - is that a separate proposal?
M is intentionally low-level. It's not supposed to be something with a rich API.

[matttproud]

Over recent quarters, I have seen folks carelessly gravitate toward using TestMain when it isn’t required (as in, there are better least-mechanism alternatives like the builtin testing facilities for the task at-hand). I think it would be useful for TestMain to have some documentation to make the low-level point you made more clear. Here is a tracer shot: “TestMain is an advanced feature and not meant for casual use. Regular testing functions will suffice for most cases.” I don’t mean to bring a point too orthogonal to the discussion, but perhaps illuminating the intended path of the user journey in documentation would provide an escape valve alternative versus expanding the API. What do you think?

[rsc]

It's certainly easy to add a sentence or two to testing.M.

[rsc]

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

[gopherbot]

Change https://golang.org/cl/334649 mentions this issue: testing: clarify in docs that TestMain is advanced

[rsc]

No change in consensus, so declined.
— rsc for the proposal review group