Conformance and Functional Tests Failing Inconsistently

[mpstefan]

Both conformance and functional tests are failing inconsistently. We need an investigation into if there is a common root cause amongst failures. We see timeouts and many other types of failures.

Acceptance

• Conformance and functional tests pass consistently on a known good PR.

Possible Causes

• Pods not ready in time
• NGINX Fails to start
• Probably not infrastructure inconsistencies

[bjee19]

After a couple days of investigation on the conformance tests, findings were limited and I could not pinpoint the cause of the flakey conformance tests. Here are some details on my findings:

After about 50-60 local test runs, with variations including: keeping the same NGF instance between test runs, deleting and restarting the NGF instance between test runs, experimental tests on/off, NGINX OSS or Plus, all test runs passed.

Compiling information from 15 or so pipeline runs, it seems like Kubernetes version, NGINX OSS or Plus, or experimental tests on/off did not have a major influence on the success of a conformance test run. Slightly more pipeline runs with experimental tests on failed, however I think that could be just because they run some more tests, increasing the likelihood of a flakey failure.

There was no pattern of a specific conformance test case failing among failed conformance test runs.

Some cases of errors that I saw:

• A conformance test would completely timeout (reaching over 10 minutes) like such: https://github.com/nginx/nginx-gateway-fabric/actions/runs/15471149959/job/43555975733 and these usually included errors where a Gateway was not being updated
• The conformance test would run fully, but certain test cases would fail like such: https://github.com/nginx/nginx-gateway-fabric/actions/runs/15471140573/job/43555965008 and these usually had failed test cases that involved a http request not getting the correct status code. (So perhaps a gateway issue, or a service issue, but it wouldn't affect other test cases)
• Seeing errors like msg: ; error: failed to preform API action, NGINX Plus API is not configured like such: https://github.com/nginx/nginx-gateway-fabric/actions/runs/15469605013/job/43550765786

Some pipeline runs would pass completely, so I am setting a current failure rate of a conformance pipeline run at 1/12 ~= 8%

There is currently some thought that a fix of a bug on the NGINX Agent side might fix some of these issues.

[bjee19]

Note: On local and pipeline runs with NGINX Plus, the conformance test still passes even if these errors exist:

msg: ; error: failed to preform API action, NGINX Plus API is not configured. However an excessive amount of these has also lead to the conformance test timing out.

Example job that has those errors, and yet still passes: https://github.com/nginx/nginx-gateway-fabric/actions/runs/15473165211/job/43562767814

[salonichf5]

After investigating for 2 days, I am not able to pin point exactly why the functional tests are failing. I did manual runs of NGF with OSS and plus a lot of times to reciprocate the issue locally but couldn't get the tests that were failing in the pipeline fail locally.

Some of the failing pipeline test failures were -

1. Upstream settings policy - https://github.com/nginx/nginx-gateway-fabric/actions/runs/15488898408/job/43610283915?pr=3470
   https://github.com/nginx/nginx-gateway-fabric/actions/runs/15496805715/job/43635663627

2. SnippetsFilter

3. Graceful recovery tests fails the most number of times - https://github.com/nginx/nginx-gateway-fabric/actions/runs/15488921174/job/43610406013?pr=3426

Upon further investigation using print statements in pipeline, graceful recovery tests fail due to upstreams not being available when failing or working traffic is being checked. I did a manual run with plus and different test scenarios in the suite but couldn't reciprocate the original error.

I have opened a PR to ignore the upstream error message to avoid issues. Once we have a bug fix from Agent team, we can remove this error message and re-verify if the issue still exists.