Skip to content

Conformance job in pipeline is not failing when tests are failing #1176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ciarams87 opened this issue Oct 23, 2023 · 0 comments · Fixed by #1177
Closed

Conformance job in pipeline is not failing when tests are failing #1176

ciarams87 opened this issue Oct 23, 2023 · 0 comments · Fixed by #1177
Assignees
@ciarams87
Labels
bug Something isn't working refined Requirements are refined and the issue is ready to be implemented. size/extra-small Estimated to be completed within a day
Milestone
v1.0.0

Comments

@ciarams87
Copy link
Contributor

ciarams87 commented Oct 23, 2023 •
edited
Loading

The conformance tests jobs are not being marked as failed in the pipeline even when there are failed tests. This is because the last command in the pod and the Make target relate to the conformance profile report and not the tests themselves, so the exit code is always 0 as long as the job completes.

To Reproduce
Open a PR with a known failed conformance test. See https://github.com/nginxinc/nginx-gateway-fabric/actions/runs/6589264877/job/17903338887?pr=1169 for an example.

Expected behavior
Conformance test job should fail when any conformance tests fail.
@ciarams87 ciarams87 mentioned this issue Oct 23, 2023
Merged
6 tasks
@ciarams87 ciarams87 self-assigned this Oct 23, 2023
@ciarams87 ciarams87 added the bug Something isn't working label Oct 23, 2023
@ciarams87 ciarams87 added this to the v1.0.0 milestone Oct 23, 2023
@mpstefan mpstefan added the refined Requirements are refined and the issue is ready to be implemented. label Oct 23, 2023
@ciarams87 ciarams87 added the size/extra-small Estimated to be completed within a day label Oct 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ciarams87 ciarams87

Labels
bug Something isn't working refined Requirements are refined and the issue is ready to be implemented. size/extra-small Estimated to be completed within a day
Projects
None yet
Milestone
v1.0.0
Development

Successfully merging a pull request may close this issue.

Fail run conformance test make target on test failure ciarams87/nginx-gateway-fabric
2 participants
@mpstefan @ciarams87