Balance unit tests across multiple threads

[weswigham]

Previously we just reserved one thread for unit tests; but they now take longer on their own than the average execution time of all the other threads - making them the critical path for testing. This change allows us to execute (and profile!) each suite separately; after an initial run to gather profiling data on the unit tests, this managed to bring my total runtime down by a few seconds (~6s-10s), and perhaps more importantly, exposes gems like this (numbers are ms):

  [ 'tsrunner-unittest://extractConstants', 46651 ],
  [ 'tsrunner-unittest://extractFunctions', 44132 ],
  [ 'tsrunner-unittest://tsc-watch program updates', 16254 ],

and in a second run, to show that these numbers seem somewhat consistent:

  [ 'tsrunner-unittest://extractConstants', 44972 ],
  [ 'tsrunner-unittest://extractFunctions', 43215 ],
  [ 'tsrunner-unittest://tsc-watch program updates', 14786 ],

Our three longest-running test suites are unittest suites, and are recently modified/created. I was wondering why test time went up so much recently, and now I know why. 🐱 If I felt comfortable skipping them, I could save almost 2 minutes of total wall-clock time (dived across 8 threads - so likely ~13s of real time) just by skipping those tests.

[sandersn]

why not add "unittest" to TestRunnerKind?

[weswigham]

We use TestRunnerKind outside the parallel runner to mean "anything for which we have a mapping to a runner class". Unfortunately, the logic doesn't generalize well to normal unit tests (as they don't conform to that API).