Report progress of individual test cases

[gaearon]

🚀 Feature Proposal

It seems like Jest currently only reports progress when a complete suite passes.
I propose that instead, it should report progress every second for individual test cases.

Otherwise, the progress output is confusing. "Tests: N" being a separate line in the output gives the impression that it's just as granular as "Test Suites: N" and will be incremented immediately when a test case passes.

Motivation

This is confusing for large/slow suites because you don't get any sense of how much time is left and whether tests are stuck for some reason.

[SimenB]

Sorta related: #6225. But this has an actual use case attached.

[gaearon]

Btw if you want to reproduce, clone facebook/prepack, run yarn and then yarn test-react --watch.

[SimenB]

We don't know the number of tests before executing the test files (and its describes). But populating that information might be possible at that time?

@aaronabramov thoughts?

[jacklinj066]

The video explains the complete coding process and is quite nice. Thank you for sharing. React Js development company in chennai

[gaearon]

I guess my mental model was that each test() or it() call goes into Jest, so Jest should be able to “ping” the watcher about its progress on every exit from an it().

[boujeepossum]

yeah, we should be able to know everything about the number of tests we have once we executed the code in the test file (but before running it)
once we stream the results it should be a one line change to implement that!

[gaearon]

Why is the total number of the tests intersting?

I want to clarify my feature request isn’t about seeing a total number of tests before they run. It’s about seeing the passed number of tests change as each of them completes.

[boujeepossum]

yeah, i agree. we don't need it.
we should just report a single test and increment a counter once it or test is done

[rickhanlonii]

+1, with this I can finish a port of the ava mini reporter to Jest

Here's the current reporter interface:

export default class BaseReporter {
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}

  // these are actually for the test suite, not individual test results
  onTestStart(test: Test) {}
  onTestResult(test: Test, testResult: TestResult, results: AggregatedResult) {}

  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

The ideal for me would be (note some of the types are either renamed of do not exist yet):

export default class BaseReporter {
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}
 
  onTestSuiteStart(suite: TestSuite) {}

  // new: called at the beginning of a .test() or .it() with the test meta info
  onTestStart(test: Test) {}

  // new: called at the end of a .test() or .it() the test meta info and result info
  onTestResult(suite: TestSuite, test: Test, testResult: TestResult, results: AggregatedResult) {}

  onTestSuiteResult(suite: TestSuite, testSuiteResult: TestSuiteResult, results: AggregatedResult) {}

  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

Until a major we could call a "test" the test suite and "assertion" an individual test (which pairs with some of the types we use):

export default class BaseReporter {
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}

  onTestStart(test: Test) {}
  
  onAssertionStart(assertion: Assertion) {} // note this type doesn't exist

  onAssertionResult(test: Test, assertion: Assertion, assertionResult: AssertionResult, results: AggregatedResult) {}

  onTestSuiteResult(test: Test, testResult: TestResult, results: AggregatedResult) {}

  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

[boujeepossum]

i'm not sure about using the word "assertion" this way.
Right now we have a huge term overload. i think almost everyone at facebook uses "test" to describe the whole file (unlike test suite that we normally use in jest).
then i normally hear the term "test case" to describe individual test or it and assertion to describe an expect or assert call.

[rickhanlonii]

Yeah that makes sense, I like "test case" and that doesn't require renaming anything. I only used "assertion" because the type of a test case result is called AssertionResult

Here's what we can go with then:

export default class BaseReporter {
  // Called at the beginning of a run
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}

  // Called at the beginning of every test file
  onTestStart(test: Test) {}
  
  // Called at the beginning of every .test or .it
  onTestCaseStart(test: Test, testCase: TestCase) {}

  // Called with the result of every .test or .it
  onTestCaseResult(test: Test, testCase: TestCase, testCaseResult: AssertionResult) {}

  // Called with the result of every test file
  onTestResult(test: Test, testResult: TestResult, results: AggregatedResult) {}

  // Called after all tests have completed
  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

[SimenB]

Could we call it Spec instead of TestCase? Matches Jasmine (https://jasmine.github.io/api/3.0/global.html#it).

FWIW the Jest docs calls a single it/test a "test", nowhere I can see is test meant to be the whole file

[gaearon]

nowhere I can see is test meant to be the whole file

👍 I think calling the whole file a "test" is a Facebook-ism, I've never seen this anywhere else. I'd love if Jest stopped using this terminology.

[boujeepossum]

makes sense! if we clean up onTestResult methods and rename them to be onTestSuiteResult we can make it consistent with documentation (and reporters output).
so pretty much what @rickhanlonii wrote earlier:

test_run - all tests/test suites in all runners
test_suite - one file
test - one it/test
assertion - one expect call