testing: TestMain should not require os.Exit

[abuchanan-nr]

TestMain is expected to call os.Exit itself. This leads to subtle issues with defer and panic that leads to much headscratching. I'm proposing that if TestMain doesn't call os.Exit, the testing framework will call it for you after TestMain returns.

Let's say you start here:

func TestMain(m *testing.M) {
  setup()
  defer teardown()
  os.Exit(m.Run())
}

Subtle bug #1: your teardown isn't running; you're making a mess of your test environment. Try this:

func TestMain(m *testing.M) {
  var exitCode int
  defer os.Exit(exitCode)

  setup()
  defer teardown()
  exitCode = m.Run()
}

Oops, setup has a panic call. Now when you run go test, it's silent..

go test .
ok  	_/Users/abuchanan/scratch/testmain_bug	0.010s

Uh, headscratch. "Why isn't go running my tests?", you ask. Oh, os.Exit is hiding my panic call. Ok, finally:

func TestMain(m *testing.M) {
  // testMain wrapper is needed to support defers and panics.
  // os.Exit will ignore those and exit silently.
  os.Exit(testMain(m))
}

func testMain(m *testing.M) int {
  setup()
  defer teardown()
  return m.Run()
}

Let's save people some headscratching (and time and effort and silly wrapper code). The testing framework probably has enough information to call os.Exit appropriately for you after TestMain returns, right? Why require users to call it?

TestMain is really useful for writing end-to-end tests, where a single setup/teardown for all the tests is important. If I break those tests into packages for organization, it would be nice not to copy that TestMain wrapper code everywhere.

[ianlancetaylor]

Related: #9917.

One possibility, that I thought was discussed before but I can't find an issue, is to change the behavior depending on the result of TestMain. If TestMain does not return a value, assume it calls os.Exit. If TestMain returns an int, then pass that int to os.Exit.

[abuchanan-nr]

#23404

[ianlancetaylor]

Ah, thanks.

[rsc]

This mistake does happen a lot, and the testing package that called TestMain and prepared the m can certainly record the result of m.Run for use in its own outer os.Exit.

Letting TestMain simply call m.Run instead of calling os.Exit(m.Run) seems OK and makes defer more useful.

I'm not as confident about alternate TestMain signatures; that seems unnecessary to solve this specific problem.

/cc @mpvl @robpike

[rsc]

@robpike and @mpvl, any thoughts?

[robpike]

No real insight from me. The topic seems covered by @ianlancetaylor's and @rsc's points.

[rsc]

It seems like if we do the "automatic os.Exit" that will fix quite a few buggy tests, and it basically eliminates the need for a TestMain that returns int. We could always revisit that and add it later, if needed, but let's start with the automatic os.Exit.

To be clear, the proposal is this. Right now, the overall test func main calls TestMain(m), and the TestMain function is expected to do two things:

1. Call m.Run() and record the result.
2. Call os.Exit with the result of m.Run.

The proposal is that:

• m.Run records its own result in an unexported field of m, and then
• if TestMain(m) returns, the outer test func main harness calls os.Exit with that code.

This would mean that implementations of TestMain that forget to call os.Exit will have it called for them. And implementations that want to explicitly avoid os.Exit in order to make defer useful can do that.

If TestMain does not call m.Run (presumably for a good reason) and returns, then the outer func main should os.Exit(0) as it does today.

Does anyone object to this plan or think it would not address the necessary use cases?

[rsc]

This seems like a likely accept.

Leaving open a week for final comments.

[bradfitz]

No new comments in the past week, so marking as accepted.