Skip to content

Commit d455f08

Browse files
sjbermansjberman
authored
NFR Test Results for NGF version 2.0.0 (#3436) (#3438)
Update with NFR test results for NGF version 2.0.0 ["oss","plus"] Auto-generated by the NFR tests workflow run 15396102440 Tests ran using Docker image tag release-2.0-rc all test(s) ran
1 parent 3dae17e commit d455f08

File tree

70 files changed

+1722
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

70 files changed

+1722
-0
lines changed

‎tests/results/dp-perf/2.0.0/2.0.0-oss.md

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Results
2+
3+
## Test environment
4+
5+
NGINX Plus: false
6+
7+
NGINX Gateway Fabric:
8+
9+
- Commit: f06b10da4a1754e253a58a31d2eff1bd404e33cd
10+
- Date: 2025-06-02T15:20:05Z
11+
- Dirty: false
12+
13+
GKE Cluster:
14+
15+
- Node count: 12
16+
- k8s version: v1.32.4-gke.1106006
17+
- vCPUs per node: 16
18+
- RAM per node: 65851340Ki
19+
- Max pods per node: 110
20+
- Zone: us-west1-b
21+
- Instance Type: n2d-standard-16
22+
23+
## Summary:
24+
25+
- Average latency slightly increased across all routing methods.
26+
27+
## Test1: Running latte path based routing
28+
29+
```text
30+
Requests [total, rate, throughput] 30000, 1000.00, 999.97
31+
Duration [total, attack, wait] 30.001s, 30s, 855.194µs
32+
Latencies [min, mean, 50, 90, 95, 99, max] 632.128µs, 875.548µs, 862.905µs, 983.833µs, 1.031ms, 1.148ms, 6.856ms
33+
Bytes In [total, mean] 4800000, 160.00
34+
Bytes Out [total, mean] 0, 0.00
35+
Success [ratio] 100.00%
36+
Status Codes [code:count] 200:30000
37+
Error Set:
38+
```
39+
40+
## Test2: Running coffee header based routing
41+
42+
```text
43+
Requests [total, rate, throughput] 30000, 999.98, 999.95
44+
Duration [total, attack, wait] 30.002s, 30.001s, 1.004ms
45+
Latencies [min, mean, 50, 90, 95, 99, max] 692.343µs, 908.751µs, 893.265µs, 1.009ms, 1.054ms, 1.176ms, 30.581ms
46+
Bytes In [total, mean] 4830000, 161.00
47+
Bytes Out [total, mean] 0, 0.00
48+
Success [ratio] 100.00%
49+
Status Codes [code:count] 200:30000
50+
Error Set:
51+
```
52+
53+
## Test3: Running coffee query based routing
54+
55+
```text
56+
Requests [total, rate, throughput] 30000, 1000.02, 999.99
57+
Duration [total, attack, wait] 30s, 29.999s, 841.484µs
58+
Latencies [min, mean, 50, 90, 95, 99, max] 695.394µs, 912.502µs, 898.684µs, 1.022ms, 1.068ms, 1.183ms, 10.847ms
59+
Bytes In [total, mean] 5070000, 169.00
60+
Bytes Out [total, mean] 0, 0.00
61+
Success [ratio] 100.00%
62+
Status Codes [code:count] 200:30000
63+
Error Set:
64+
```
65+
66+
## Test4: Running tea GET method based routing
67+
68+
```text
69+
Requests [total, rate, throughput] 30000, 1000.01, 999.98
70+
Duration [total, attack, wait] 30.001s, 30s, 955.507µs
71+
Latencies [min, mean, 50, 90, 95, 99, max] 685.791µs, 902.297µs, 888.939µs, 1.01ms, 1.057ms, 1.166ms, 9.243ms
72+
Bytes In [total, mean] 4740000, 158.00
73+
Bytes Out [total, mean] 0, 0.00
74+
Success [ratio] 100.00%
75+
Status Codes [code:count] 200:30000
76+
Error Set:
77+
```
78+
79+
## Test5: Running tea POST method based routing
80+
81+
```text
82+
Requests [total, rate, throughput] 29999, 1000.00, 999.97
83+
Duration [total, attack, wait] 30s, 29.999s, 933.986µs
84+
Latencies [min, mean, 50, 90, 95, 99, max] 666.013µs, 916.212µs, 887.892µs, 1.011ms, 1.06ms, 1.191ms, 207.34ms
85+
Bytes In [total, mean] 4739842, 158.00
86+
Bytes Out [total, mean] 0, 0.00
87+
Success [ratio] 100.00%
88+
Status Codes [code:count] 200:29999
89+
Error Set:
90+
```

‎tests/results/dp-perf/2.0.0/2.0.0-plus.md

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Results
2+
3+
## Test environment
4+
5+
NGINX Plus: true
6+
7+
NGINX Gateway Fabric:
8+
9+
- Commit: f06b10da4a1754e253a58a31d2eff1bd404e33cd
10+
- Date: 2025-06-02T15:20:05Z
11+
- Dirty: false
12+
13+
GKE Cluster:
14+
15+
- Node count: 12
16+
- k8s version: v1.32.4-gke.1106006
17+
- vCPUs per node: 16
18+
- RAM per node: 65851340Ki
19+
- Max pods per node: 110
20+
- Zone: us-west1-b
21+
- Instance Type: n2d-standard-16
22+
23+
## Summary:
24+
25+
- Average latency slightly increased across all routing methods.
26+
27+
## Test1: Running latte path based routing
28+
29+
```text
30+
Requests [total, rate, throughput] 30000, 999.97, 999.82
31+
Duration [total, attack, wait] 30.005s, 30.001s, 4.397ms
32+
Latencies [min, mean, 50, 90, 95, 99, max] 674.871µs, 1.095s, 991.559µs, 5.087s, 6.518s, 7.551s, 7.768s
33+
Bytes In [total, mean] 4770000, 159.00
34+
Bytes Out [total, mean] 0, 0.00
35+
Success [ratio] 100.00%
36+
Status Codes [code:count] 200:30000
37+
Error Set:
38+
```
39+
40+
## Test2: Running coffee header based routing
41+
42+
```text
43+
Requests [total, rate, throughput] 30000, 1000.04, 1000.00
44+
Duration [total, attack, wait] 30s, 29.999s, 1.129ms
45+
Latencies [min, mean, 50, 90, 95, 99, max] 658.096µs, 960.704µs, 943.39µs, 1.091ms, 1.144ms, 1.34ms, 4.704ms
46+
Bytes In [total, mean] 4800000, 160.00
47+
Bytes Out [total, mean] 0, 0.00
48+
Success [ratio] 100.00%
49+
Status Codes [code:count] 200:30000
50+
Error Set:
51+
```
52+
53+
## Test3: Running coffee query based routing
54+
55+
```text
56+
Requests [total, rate, throughput] 30000, 1000.00, 999.97
57+
Duration [total, attack, wait] 30.001s, 30s, 903.354µs
58+
Latencies [min, mean, 50, 90, 95, 99, max] 666.726µs, 962.973µs, 938.524µs, 1.094ms, 1.153ms, 1.375ms, 16.612ms
59+
Bytes In [total, mean] 5040000, 168.00
60+
Bytes Out [total, mean] 0, 0.00
61+
Success [ratio] 100.00%
62+
Status Codes [code:count] 200:30000
63+
Error Set:
64+
```
65+
66+
## Test4: Running tea GET method based routing
67+
68+
```text
69+
Requests [total, rate, throughput] 30000, 1000.04, 1000.01
70+
Duration [total, attack, wait] 30s, 29.999s, 905.444µs
71+
Latencies [min, mean, 50, 90, 95, 99, max] 664.584µs, 941.233µs, 924.206µs, 1.079ms, 1.138ms, 1.324ms, 6.779ms
72+
Bytes In [total, mean] 4710000, 157.00
73+
Bytes Out [total, mean] 0, 0.00
74+
Success [ratio] 100.00%
75+
Status Codes [code:count] 200:30000
76+
Error Set:
77+
```
78+
79+
## Test5: Running tea POST method based routing
80+
81+
```text
82+
Requests [total, rate, throughput] 30000, 1000.03, 999.99
83+
Duration [total, attack, wait] 30s, 29.999s, 1.099ms
84+
Latencies [min, mean, 50, 90, 95, 99, max] 651.328µs, 944.335µs, 927.829µs, 1.081ms, 1.136ms, 1.29ms, 13.497ms
85+
Bytes In [total, mean] 4710000, 157.00
86+
Bytes Out [total, mean] 0, 0.00
87+
Success [ratio] 100.00%
88+
Status Codes [code:count] 200:30000
89+
Error Set:
90+
```

‎tests/results/reconfig/2.0.0/2.0.0-oss.md

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
# Results
2+
3+
## Test environment
4+
5+
NGINX Plus: false
6+
7+
NGINX Gateway Fabric:
8+
9+
- Commit: f06b10da4a1754e253a58a31d2eff1bd404e33cd
10+
- Date: 2025-06-02T15:20:05Z
11+
- Dirty: false
12+
13+
GKE Cluster:
14+
15+
- Node count: 12
16+
- k8s version: v1.32.4-gke.1106006
17+
- vCPUs per node: 16
18+
- RAM per node: 65851340Ki
19+
- Max pods per node: 110
20+
- Zone: us-west1-b
21+
- Instance Type: n2d-standard-16
22+
23+
## Summary:
24+
25+
- Expectations of the test suite have changed a bit due to the new architecture.
26+
- Time to ready generally increases due to provisioning NGINX.
27+
- Seeing some jumbled configuration issues, this has been brought to the attention of the NGINX Agent team.
28+
29+
## Test 1: Resources exist before startup - NumResources 30
30+
31+
### Time to Ready
32+
33+
Time To Ready Description: From when NGF starts to when the NGINX configuration is fully configured
34+
- TimeToReadyTotal: 15s
35+
36+
### Event Batch Processing
37+
38+
- Event Batch Total: 10
39+
- Event Batch Processing Average Time: 3ms
40+
- Event Batch Processing distribution:
41+
- 500.0ms: 10
42+
- 1000.0ms: 10
43+
- 5000.0ms: 10
44+
- 10000.0ms: 10
45+
- 30000.0ms: 10
46+
- +Infms: 10
47+
48+
### NGINX Error Logs
49+
50+
## Test 1: Resources exist before startup - NumResources 150
51+
52+
### Time to Ready
53+
54+
Time To Ready Description: From when NGF starts to when the NGINX configuration is fully configured
55+
- TimeToReadyTotal: 9s
56+
57+
### Event Batch Processing
58+
59+
- Event Batch Total: 10
60+
- Event Batch Processing Average Time: 6ms
61+
- Event Batch Processing distribution:
62+
- 500.0ms: 10
63+
- 1000.0ms: 10
64+
- 5000.0ms: 10
65+
- 10000.0ms: 10
66+
- 30000.0ms: 10
67+
- +Infms: 10
68+
69+
### NGINX Error Logs
70+
71+
## Test 2: Start NGF, deploy Gateway, wait until NGINX agent instance connects to NGF, create many resources attached to GW - NumResources 30
72+
73+
### Time to Ready
74+
75+
Time To Ready Description: From when NGINX receives the first configuration created by NGF to when the NGINX configuration is fully configured
76+
- TimeToReadyTotal: 21s
77+
78+
### Event Batch Processing
79+
80+
- Event Batch Total: 315
81+
- Event Batch Processing Average Time: 11ms
82+
- Event Batch Processing distribution:
83+
- 500.0ms: 315
84+
- 1000.0ms: 315
85+
- 5000.0ms: 315
86+
- 10000.0ms: 315
87+
- 30000.0ms: 315
88+
- +Infms: 315
89+
90+
### NGINX Error Logs
91+
2025/06/02 16:12:16 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:666
92+
2025/06/02 16:12:16 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:1004
93+
2025/06/02 16:12:17 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:1113
94+
2025/06/02 16:12:20 [emerg] 8#8: pread() returned only 0 bytes instead of 4086 in /etc/nginx/conf.d/http.conf:715
95+
96+
## Test 2: Start NGF, deploy Gateway, wait until NGINX agent instance connects to NGF, create many resources attached to GW - NumResources 150
97+
98+
### Time to Ready
99+
100+
Time To Ready Description: From when NGINX receives the first configuration created by NGF to when the NGINX configuration is fully configured
101+
- TimeToReadyTotal: 109s
102+
103+
### Event Batch Processing
104+
105+
- Event Batch Total: 1445
106+
- Event Batch Processing Average Time: 18ms
107+
- Event Batch Processing distribution:
108+
- 500.0ms: 1445
109+
- 1000.0ms: 1445
110+
- 5000.0ms: 1445
111+
- 10000.0ms: 1445
112+
- 30000.0ms: 1445
113+
- +Infms: 1445
114+
115+
### NGINX Error Logs
116+
2025/06/02 16:16:12 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:197
117+
2025/06/02 16:16:13 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:1004
118+
2025/06/02 16:16:14 [emerg] 8#8: unknown directive "ts/ssl_keypair_reconfig_cafe-secret.pem" in /etc/nginx/conf.d/http.conf:189
119+
2025/06/02 16:16:15 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:1974
120+
2025/06/02 16:16:17 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:2639
121+
2025/06/02 16:16:18 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:3282
122+
2025/06/02 16:16:18 [emerg] 8#8: pread() returned only 0 bytes instead of 4089 in /etc/nginx/conf.d/http.conf:2416
123+
2025/06/02 16:16:21 [emerg] 8#8: pread() returned only 0 bytes instead of 4069 in /etc/nginx/conf.d/http.conf:193
124+
2025/06/02 16:16:21 [emerg] 8#8: pread() returned only 0 bytes instead of 4086 in /etc/nginx/conf.d/http.conf:905
125+
2025/06/02 16:16:25 [emerg] 8#8: invalid number of arguments in "zone" directive in /etc/nginx/conf.d/http.conf:5862
126+
2025/06/02 16:16:26 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:6949
127+
2025/06/02 16:16:27 [emerg] 8#8: pread() returned only 0 bytes instead of 4093 in /etc/nginx/conf.d/http.conf:2421
128+
2025/06/02 16:16:27 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:7315
129+
2025/06/02 16:16:27 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:7494
130+
2025/06/02 16:16:28 [emerg] 8#8: pread() returned only 0 bytes instead of 4081 in /etc/nginx/conf.d/http.conf:3875
131+
2025/06/02 16:16:28 [emerg] 8#8: pread() returned only 0 bytes instead of 4095 in /etc/nginx/conf.d/http.conf:4364
132+
2025/06/02 16:16:28 [emerg] 8#8: pread() returned only 0 bytes instead of 4093 in /etc/nginx/conf.d/http.conf:4847
133+
2025/06/02 16:16:29 [emerg] 8#8: pread() returned only 0 bytes instead of 4094 in /etc/nginx/conf.d/http.conf:632
134+
2025/06/02 16:16:29 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:8514
135+
2025/06/02 16:16:29 [emerg] 8#8: pread() returned only 0 bytes instead of 4085 in /etc/nginx/conf.d/http.conf:7409
136+
2025/06/02 16:16:29 [emerg] 8#8: pread() returned only 0 bytes instead of 4096 in /etc/nginx/conf.d/http.conf:6299
137+
2025/06/02 16:16:31 [emerg] 8#8: pread() returned only 0 bytes instead of 4084 in /etc/nginx/conf.d/http.conf:6343
138+
2025/06/02 16:16:33 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:10970
139+
2025/06/02 16:16:35 [emerg] 8#8: pread() returned only 0 bytes instead of 4095 in /etc/nginx/conf.d/http.conf:4495
140+
2025/06/02 16:16:35 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:11951
141+
2025/06/02 16:16:36 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:12438
142+
2025/06/02 16:16:37 [emerg] 8#8: pread() returned only 0 bytes instead of 4095 in /etc/nginx/conf.d/http.conf:8234
143+
2025/06/02 16:16:39 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:13878
144+
2025/06/02 16:16:39 [emerg] 8#8: pread() returned only 0 bytes instead of 4076 in /etc/nginx/conf.d/http.conf:2329
145+
2025/06/02 16:16:41 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:15054
146+
2025/06/02 16:16:41 [emerg] 8#8: pread() returned only 0 bytes instead of 4095 in /etc/nginx/conf.d/http.conf:9862
147+
2025/06/02 16:16:42 [emerg] 8#8: pread() returned only 0 bytes instead of 4079 in /etc/nginx/conf.d/http.conf:14393
148+
2025/06/02 16:16:42 [emerg] 8#8: pread() returned only 0 bytes instead of 2780 in /etc/nginx/conf.d/http.conf:15535
149+
2025/06/02 16:16:43 [emerg] 8#8: pread() returned only 0 bytes instead of 4083 in /etc/nginx/conf.d/http.conf:5118
150+
2025/06/02 16:16:43 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:16167

0 commit comments

Comments
 (0)