[ci] Handle the case where all reported tests pass but the build is still a failure #120264
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…till a failure In this build: https://buildkite.com/llvm-project/github-pull-requests/builds/126961 The builds actually failed, probably because prerequisite of a test suite failed to build. However they still ran other tests and all those passed. This meant that the test reports were green even though the build was red. On some level this is technically correct, but's very misleading in practice. So I've also passed the build script's return code, as it was when we entered the on exit handler, to the generator. So that when this happens again, the report will not list any test failures but will draw the viewer's attention to the overall failure. There will be a link in the report to the build's log file, so the next step to investigate is clear. It would be nice to say "tests failed and there was some other build error", but we cannot tell what the non-zero return code was caused by. Could be either.
|
This is all done within the "if we have a buildkite agent", so it should not disrupt work on the new system. It's getting a bit complex, but (famous last words) I think this is the last combination it needs to handle, as shown in the table. |
|
Pretty sure I'm not the right reviewer here... Maybe @lnihlen is? |
lnihlen
reviewed
Jan 10, 2025
There was a problem hiding this comment.
Thanks for working this up. I'd like @boomanaiden154 to verify this keeps the new infra happy too, and had a question about fragility in one of the tests, but otherwise LGTM
boomanaiden154
approved these changes
Jan 10, 2025
kazutakahirata
pushed a commit
to kazutakahirata/llvm-project
that referenced
this pull request
Jan 13, 2025
…till a failure (llvm#120264) In this build: https://buildkite.com/llvm-project/github-pull-requests/builds/126961 The builds actually failed, probably because prerequisite of a test suite failed to build. However they still ran other tests and all those passed. This meant that the test reports were green even though the build was red. On some level this is technically correct, but it is very misleading in practice. So I've also passed the build script's return code, as it was when we entered the on exit handler, to the generator, so that when this happens again, the report will draw the viewer's attention to the overall failure. There will be a link in the report to the build's log file, so the next step to investigate is clear. It would be nice to say "tests failed and there was some other build error", but we cannot tell what the non-zero return code was caused by. Could be either. The script handles the following situations now: | Have Result Files? | Tests reported failed? | Return code | Report | |--------------------|------------------------|-------------|-----------------------------------------------------------------------------| | Yes | No | 0 | Success style report. | | Yes | Yes | 0 | Shouldn't happen, but if it did, failure style report showing the failures. | | Yes | No | 1 | Failure style report, showing no failures but noting that the build failed. | | Yes | Yes | 1 | Failure style report, showing the test failures. | | No | ? | 0 | No test report, success shown in the normal build display. | | No | ? | 1 | No test report, failure shown in the normal build display. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this build:
https://buildkite.com/llvm-project/github-pull-requests/builds/126961
The builds actually failed, probably because prerequisite of a test suite failed to build.
However they still ran other tests and all those passed. This meant that the test reports were green even though the build was red. On some level this is technically correct, but it is very misleading in practice.
So I've also passed the build script's return code, as it was when we entered the on exit handler, to the generator, so that when this happens again, the report will draw the viewer's attention to the overall failure. There will be a link in the report to the build's log file, so the next step to investigate is clear.
It would be nice to say "tests failed and there was some other build error", but we cannot tell what the non-zero return code was caused by. Could be either.
The script handles the following situations now: