Cancel test run doesn't seem to work

[rzhao271]

Testing #21494

I changed the parameterized test to go from 1 to 10k to 1 to 1k, and then pressed the run button, expecting the test changes to be picked up. However, I noticed the parameterized test was going past 1k because I had the tests expanded while the tests were running. I tried to cancel, but pressing cancel did nothing. I notice now that pressing cancel seems to work after the parameterized test finishes running. I'm wondering if the token can be checked per iteration of a parameterized test?

[bhavyaus]

Noticed the same behavior with vscode crashing after I hit Cancel Test Run in succession. I changed the parametrized test to go from 1 to 10k.

[eleanorjboyd]

On further testing with 10k tests I have found that the cancelation token is called immediately once the stop button is clicked followed by the TestRun object getting set to testRun.end(). I am not sure entirely about the delay, since I see one too and it seems to occur following the cancelation token call. When I observe it locally my extension logs print Setting the state of test "param_folder/test_parameterized.py::test_odd_even[93]" is a no-op after the run ends. for every test that was not finished and instead canceled. This error comes from here. I do notice that it spins for as long as it takes to print these logs which then stop spinning once it has gotten to the final one log but this is likely specific for the debugging mode I was in. @connor4312 do you have any thoughts?

This is at parity with the current testing infrastructure (rewrite vs non-rewrite code) so it is less urgent to get it out for this release but would be good to get to the bottom of. Thanks!

[connor4312]

It seems like the console logs are pretty slow, but there's also a UI stall when ending all those tests. I will have a PR for the latter, and the Python extension should avoid setting states after calling end()

[eleanorjboyd]

@connor4312 I thought we did avoid setting states after end() was called. Was there an instance you saw this was not the case?

[connor4312]

The console log "Setting the state of test "param_folder/test_parameterized.py::test_odd_even[93]" is a no-op after the run ends" is triggered by setting the state after end gets called. I also observed those logs when I was looking into the issue.