Skip to content

Test reporting should consistently count and report test failures, not compiler errors #4383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

Blaisorblade
Copy link
Contributor

While individual tests should compare actual and expected numbers of errors, the overall reporting is meant to report failures.
Discovered while investigating the root cause of #4379 (where problems arose from a similar confusion) — so the PR is built on top of that (and intended to be merged separately).

@Blaisorblade Blaisorblade requested a review from allanrenucci May 1, 2018 16:43
Right now we report the number of "errors", that is compiler errors including
the expected ones, but we should instead use the test "failures", that is the
unexpected behaviors.

Also make errorCount private.
This small issue swaps labels in the error output and complicated fixing
failures.
We can't use `_failures.size` since it's a Set. Using a `Seq` for now would be
pointless, since most elements are just `Generic` failures, but would have a
point if we wanted to show info on those failures in the final log.
This output refers to the number of CompilationTest instances with failures.
@Blaisorblade Blaisorblade force-pushed the test-replace-errors-by-failures branch from 033f85b to 4300c07 Compare May 1, 2018 16:47
@Blaisorblade Blaisorblade self-assigned this May 1, 2018
@Blaisorblade Blaisorblade merged commit 9b42731 into scala:master May 1, 2018
@Blaisorblade Blaisorblade deleted the test-replace-errors-by-failures branch May 1, 2018 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants