Skip to content

Conformance job in pipeline is not failing when tests are failing #1176

Closed
@ciarams87

Description

@ciarams87

The conformance tests jobs are not being marked as failed in the pipeline even when there are failed tests. This is because the last command in the pod and the Make target relate to the conformance profile report and not the tests themselves, so the exit code is always 0 as long as the job completes.

To Reproduce
Open a PR with a known failed conformance test. See https://github.com/nginxinc/nginx-gateway-fabric/actions/runs/6589264877/job/17903338887?pr=1169 for an example.

Expected behavior
Conformance test job should fail when any conformance tests fail.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingrefinedRequirements are refined and the issue is ready to be implemented.size/extra-smallEstimated to be completed within a day

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions