bpo-29512: Add test.bisect, bisect failing tests

[vstinner]

Add a new "python3 -m test.bisect" tool to bisect failing tests.

It can be used to find which test method(s) leak references, leak
files, etc.

[vstinner]

Ok, here is a first complete implementation of my "bisect" tool for Python tests. It's likely incomplete, but at least it worked well when I needed it to find a reference leak. It's especially useful for me to find a leak on Windows, where development is harder for me ;-)

[vstinner]

"# FIXME: document that following arguments are test arguments"

I failed to find the right argparse option to say that all unknown options must go into test_args, without having to declare the 20+ options of regrtest.

The code works. It's just an issue with --help which doesn't show "test_args" in the usage line.

[vstinner]

I wrote the tool to find the failing method leaking references. The tool can skip a failing test when at least 2 tests fail. Since the tool takes a random sample of tests at each iteration, it's likely that running the tool gives two different results when multiple tests fail.

It might be possible to make the tool smarter to validate that skipped tests don't fail, but I'm not sure that it's worth it, at least for the first version of the tool. It's always possible to enhance such tool step by step ;-)

[vstinner]

More concrete example of multiple failing tests. Right now, at least 3 methods of test_code leak references. Each run of test.bisect randomly displays one these failing tests:

test.test_code.CoExtra.test_free_different_thread
test.test_code.CoExtra.test_get_set
test.test_code.CoExtra.test_free_called

For me, it's ok, I understand how it works ;-)

[1st1]

The code looks fine. I'm big +1 on having more tools to debug ref leaks knowing first hand how hard it is to find such regressions. Let's merge it and work/improve it.

[vstinner]

Let's merge it and work/improve it.

Since I also proposed the same development method: ok, let's do that ;-)