Skip to content

Commit 2d033d5

Browse files
bjee19bjee19
authored
Add 1.3 reconfig test results (#2114)
Problem: Need to run reconfig test results for 1.3 release Solution: Ran the test. Testing: Ran the test.
1 parent 66318f2 commit 2d033d5

File tree

2 files changed

+113
-3
lines changed

2 files changed

+113
-3
lines changed

‎tests/reconfig/results/1.3.0/1.3.0.md

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Reconfiguration testing Results
2+
3+
<!-- TOC -->
4+
- [Reconfiguration testing Results](#reconfiguration-testing-results)
5+
- [Summary](#summary)
6+
- [Test environment](#test-environment)
7+
- [Results Tables](#results-tables)
8+
- [NGINX Reloads and Time to Ready](#nginx-reloads-and-time-to-ready)
9+
- [Event Batch Processing](#event-batch-processing)
10+
- [NumResources to Total Resources](#numresources-to-total-resources)
11+
- [Observations](#observations)
12+
- [Future Improvements](#future-improvements)
13+
<!-- TOC -->
14+
15+
## Summary
16+
17+
- Due to fix https://github.com/nginxinc/nginx-gateway-fabric/issues/1107, time to ready, reload time, and event batch processing
18+
time increased for all 150 resource tests.
19+
- For all 30 resource tests, results were mostly consistent to prior results.
20+
21+
## Test environment
22+
23+
GKE cluster:
24+
25+
- Node count: 3
26+
- Instance Type: e2-medium
27+
- k8s version: 1.28.9-gke.1000000
28+
- Zone: us-central1-c
29+
- Total vCPUs: 6
30+
- Total RAM: 12GB
31+
- Max pods per node: 110
32+
33+
NGF deployment:
34+
35+
- NGF version: edge - git commit 7c9bf23ed89861c9ce7b725f2c1686f4c24ef2f9
36+
- NGINX OSS Version: 1.27.0
37+
- NGINX Plus Version: R32
38+
39+
## Results Tables
40+
41+
### NGINX Reloads and Time to Ready
42+
43+
#### OSS
44+
45+
| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
46+
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
47+
| 1 | 30 | 2 | <1 | 2 | 190 | 100% | 100% |
48+
| 1 | 150 | 2 | <1 | 2 | 542 | 50% | 100% |
49+
| 2 | 30 | 37 | <1 | 94 | 169 | 100% | 100% |
50+
| 2 | 150 | 204 | <1 | 387 | 326 | 88% | 100% |
51+
| 3 | 30 | <1 | <1 | 94 | 129 | 100% | 100% |
52+
| 3 | 150 | <1 | <1 | 454 | 130 | 100% | 100% |
53+
54+
#### Plus
55+
56+
| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | <= 500ms | <= 1000ms |
57+
|-------------|--------------|----------------------|--------------------------|---------------|----------------------------|----------|-----------|
58+
| 1 | 30 | 1 | <1 | 2 | 220.5 | 100% | 100% |
59+
| 1 | 150 | 1.5 | <1 | 2 | 528.5 | 50% | 100% |
60+
| 2 | 30 | 41 | <1 | 94 | 176.8 | 100% | 100% |
61+
| 2 | 150 | 199 | <1 | 391 | 320.56 | 94.1% | 100% |
62+
| 3 | 30 | <1 | <1 | 94 | 128.5 | 100% | 100% |
63+
| 3 | 150 | <1 | <1 | 454 | 129.2 | 100% | 100% |
64+
65+
### Event Batch Processing
66+
67+
#### OSS
68+
69+
| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms |
70+
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|------------|
71+
| 1 | 30 | 5 | 726.6 | 80% | 80% | 100% | 100% | 100% |
72+
| 1 | 150 | 5 | 4457 | 40% | 80% | 80% | 80% | 100% |
73+
| 2 | 30 | 371 | 59.5 | 99.7% | 100% | 100% | 100% | 100% |
74+
| 2 | 150 | 1742 | 93.5 | 92.9% | 99.99% | 100% | 100% | 100% |
75+
| 3 | 30 | 370 | 43.9 | 99.85% | 99.85% | 100% | 100% | 100% |
76+
| 3 | 150 | 1810 | 44.8 | 99.99% | 99.99% | 99.99% | 100% | 100% |
77+
78+
#### Plus
79+
80+
| Test number | NumResources | Event Batch Total | Event Batch Processing avg time (ms) | <= 500ms | <= 1000ms | <= 5000ms | <= 10000ms | <= 30000ms |
81+
|-------------|--------------|-------------------|--------------------------------------|----------|-----------|-----------|------------|--------------|
82+
| 1 | 30 | 6 | 84 | 100% | 100% | 100% | 100% | 100% |
83+
| 1 | 150 | 5 | 4544.3 | 40% | 80% | 80% | 80% | 100% |
84+
| 2 | 30 | 370 | 59.1 | 100% | 100% | 100% | 100% | 100% |
85+
| 2 | 150 | 1747 | 93.2 | 94.1% | 99.99% | 100% | 100% | 100% |
86+
| 3 | 30 | 370 | 41.33 | 99.99% | 99.99% | 100% | 100% | 100% |
87+
| 3 | 150 | 1809 | 44.88 | 99.99% | 99.99% | 99.99% | 99.99% | 100% |
88+
89+
## NumResources to Total Resources
90+
91+
| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Total Resources |
92+
|--------------|----------|---------|-----------------|------------|------------------|----------------------|------------|-----------------|
93+
| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | <total> |
94+
| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 244 |
95+
| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 1204 |
96+
97+
## Observations
98+
99+
1. Reload time and time to ready have increased in 150 resource tests. This is probably due, in part, to the fix of https://github.com/nginxinc/nginx-gateway-fabric/issues/1107 causing the prior
100+
test to only attach 2x of the HTTPRoutes while this test attaches all of them. In the 30 resource tests, results were mostly consistent to prior results.
101+
102+
2. Event batch processing time increased notably in the 150 resource tests, probably for the same reason mentioned in observation #1.
103+
In the 30 resource tests, results were mostly consistent to prior results.
104+
105+
3. No errors in the logs.
106+
107+
108+
## Future Improvements
109+
110+
None.

‎tests/reconfig/setup.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ The following cluster will be sufficient:
4343

4444
```console
4545
helm install my-release oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --version 0.0.0-edge \
46-
--create-namespace --wait -n nginx-gateway --set nginxGateway.config.logging.level=debug
46+
--create-namespace --wait -n nginx-gateway --set nginxGateway.productTelemetry.enable=false
4747
```
4848

4949
4. Run tests:
@@ -67,7 +67,7 @@ The following cluster will be sufficient:
6767
Note: You can expose metrics by running the below snippet and then navigating to `127.0.0.1:9113/metrics`:
6868

6969
```console
70-
GW_POD=$(k get pods -n nginx-gateway | sed -n '2s/^\([^[:space:]]*\).*$/\1/p')
70+
GW_POD=$(kubectl get pods -n nginx-gateway | sed -n '2s/^\([^[:space:]]*\).*$/\1/p')
7171
kubectl port-forward $GW_POD -n nginx-gateway 9113:9113 &
7272
```
7373

@@ -105,7 +105,7 @@ The following cluster will be sufficient:
105105
2. Run the provided script with the required number of resources,
106106
e.g. `cd scripts && bash create-resources-routes-last.sh 30`. The script will deploy backend apps and services,
107107
wait 60 seconds for them to be ready, and deploy 1 Gateway, 1 Secret, 1 RefGrant, and HTTPRoutes at the same time.
108-
3. Measure TimeToReadyTotal as the time it takes from NGF receiving the first HTTPRoute resource update -> final
108+
3. Measure TimeToReadyTotal as the time it takes from NGF receiving the first HTTPRoute resource update (logs will say "reconciling") -> final
109109
config written and NGINX reloaded. Measure the other results as described in steps 6-7 of the [Setup](#setup) section.
110110

111111
### Test 3: Start NGF, create many resources attached to a Gateway, deploy the Gateway

0 commit comments

Comments
 (0)