diff --git a/tests/dataplane-performance/manifests/cafe-routes.yaml b/tests/dataplane-performance/manifests/cafe-routes.yaml new file mode 100644 index 0000000000..95d0252842 --- /dev/null +++ b/tests/dataplane-performance/manifests/cafe-routes.yaml @@ -0,0 +1,60 @@ +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: HTTPRoute +metadata: + name: coffee +spec: + parentRefs: + - name: cafe + hostnames: + - cafe.example.com + rules: + - matches: + - path: + type: PathPrefix + value: /latte + backendRefs: + - name: coffee-svc + port: 80 + - matches: + - path: + type: PathPrefix + value: /coffee + headers: + - name: version + value: v2 + - path: + type: PathPrefix + value: /coffee + queryParams: + - name: TEST + value: v2 + backendRefs: + - name: coffee-svc + port: 80 +--- +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: HTTPRoute +metadata: + name: tea +spec: + parentRefs: + - name: cafe + hostnames: + - cafe.example.com + rules: + - matches: + - path: + type: PathPrefix + value: /tea + method: POST + backendRefs: + - name: coffee-svc + port: 80 + - matches: + - path: + type: PathPrefix + value: /tea + method: GET + backendRefs: + - name: coffee-svc + port: 80 diff --git a/tests/dataplane-performance/manifests/coffee.yaml b/tests/dataplane-performance/manifests/coffee.yaml new file mode 100644 index 0000000000..d79b9dd36a --- /dev/null +++ b/tests/dataplane-performance/manifests/coffee.yaml @@ -0,0 +1,32 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: coffee +spec: + replicas: 1 + selector: + matchLabels: + app: coffee + template: + metadata: + labels: + app: coffee + spec: + containers: + - name: coffee + image: nginxdemos/nginx-hello:plain-text + ports: + - containerPort: 8080 +--- +apiVersion: v1 +kind: Service +metadata: + name: coffee-svc +spec: + ports: + - port: 80 + targetPort: 8080 + protocol: TCP + name: http + selector: + app: coffee diff --git a/tests/dataplane-performance/manifests/gateway.yaml b/tests/dataplane-performance/manifests/gateway.yaml new file mode 100644 index 0000000000..1d029ef08a --- /dev/null +++ b/tests/dataplane-performance/manifests/gateway.yaml @@ -0,0 +1,10 @@ +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: cafe +spec: + gatewayClassName: nginx + listeners: + - name: http + port: 80 + protocol: HTTP diff --git a/tests/dataplane-performance/results/1.0.0/1.0.0.md b/tests/dataplane-performance/results/1.0.0/1.0.0.md new file mode 100644 index 0000000000..40579b2d84 --- /dev/null +++ b/tests/dataplane-performance/results/1.0.0/1.0.0.md @@ -0,0 +1,123 @@ +# Results + +## Test environment + +GKE cluster: + +- Node count: 3 +- Instance Type: e2-medium +- k8s version: 1.27.4-gke.900 +- Zone: europe-west2-b +- Total vCPUs: 6 +- Total RAM: 12GB +- Max pods per node: 110 + +Test VM: + +- Instance Type: e2-medium +- Zone: europe-west2-b +- vCPUS: 2 +- RAM: 4GB + +NGF deployment: + +- NGF version: edge - git commit a41e9e46d72788bceea8ba44e01d4062afec75fd +- NGINX Version: 1.25.2 + +## Test1: Running latte path based routing + +```console +Running 30s test @ http://cafe.example.com/latte + 2 threads and 10 connections + Thread Stats Avg Stdev Max +/- Stdev + Latency 6.29ms 7.85ms 182.06ms 88.67% + Req/Sec 1.11k 588.11 2.31k 59.83% + Latency Distribution + 50% 3.44ms + 75% 7.09ms + 90% 15.33ms + 99% 31.86ms + 66017 requests in 30.02s, 23.42MB read +Requests/sec: 2199.43 +Transfer/sec: 799.00KB +``` + +## Test2: Running coffee header based routing + +```console +Running 30s test @ http://cafe.example.com/coffee + 2 threads and 10 connections + Thread Stats Avg Stdev Max +/- Stdev + Latency 31.95ms 37.10ms 243.16ms 89.07% + Req/Sec 224.25 238.65 1.83k 90.32% + Latency Distribution + 50% 14.31ms + 75% 43.74ms + 90% 76.70ms + 99% 171.95ms + 12913 requests in 30.03s, 4.59MB read +Requests/sec: 430.05 +Transfer/sec: 156.65KB +``` + +## Test3: Running coffee query based routing + +```console +Running 30s test @ http://cafe.example.com/coffee?TEST=v2 + 2 threads and 10 connections + Thread Stats Avg Stdev Max +/- Stdev + Latency 29.26ms 43.85ms 457.05ms 82.26% + Req/Sec 632.30 464.79 2.07k 60.43% + Latency Distribution + 50% 4.45ms + 75% 45.84ms + 90% 103.14ms + 99% 145.53ms + 37324 requests in 30.02s, 13.56MB read +Requests/sec: 1243.13 +Transfer/sec: 462.53KB +``` + +## Test4: Running tea GET method based routing + +```console +Running 30s test @ http://cafe.example.com/tea + 2 threads and 10 connections + Thread Stats Avg Stdev Max +/- Stdev + Latency 27.19ms 39.40ms 182.48ms 81.49% + Req/Sec 706.49 416.90 1.85k 65.42% + Latency Distribution + 50% 4.06ms + 75% 42.24ms + 90% 99.21ms + 99% 133.34ms + 41265 requests in 30.04s, 14.56MB read +Requests/sec: 1373.59 +Transfer/sec: 496.31KB +``` + +## Test5: Running tea POST method based routing + +```console +Running 30s test @ http://cafe.example.com/tea + 2 threads and 10 connections + Thread Stats Avg Stdev Max +/- Stdev + Latency 27.32ms 39.49ms 208.66ms 81.40% + Req/Sec 685.47 491.33 2.08k 58.43% + Latency Distribution + 50% 4.19ms + 75% 42.40ms + 90% 99.38ms + 99% 133.25ms + 40723 requests in 30.03s, 14.37MB read +Requests/sec: 1356.10 +Transfer/sec: 489.99KB +``` + +## Observations + +- Path based routing is the most performant of the routing methods +- The njs based methods are less performant: + - Header based routing is the least performant of the routing methods, by a factor of roughly 5 compared to the path + based routing. + - The query and method based routing methods are approximately 2/3s as performant as the path based routing. diff --git a/tests/dataplane-performance/scripts/post.lua b/tests/dataplane-performance/scripts/post.lua new file mode 100644 index 0000000000..ab5401b28c --- /dev/null +++ b/tests/dataplane-performance/scripts/post.lua @@ -0,0 +1 @@ +wrk.method = "POST" diff --git a/tests/dataplane-performance/scripts/wrk-latency.sh b/tests/dataplane-performance/scripts/wrk-latency.sh new file mode 100644 index 0000000000..21cf27b8c6 --- /dev/null +++ b/tests/dataplane-performance/scripts/wrk-latency.sh @@ -0,0 +1,26 @@ +#!/bin/bash + +SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) + +echo -e "# Results\n" + +echo -e "## Test1: Running latte path based routing\n" +echo -e '```console' +wrk -t2 -c10 -d30 http://cafe.example.com/latte --latency +echo -e '```' +echo -e "\n## Test2: Running coffee header based routing\n" +echo -e '```console' +wrk -t2 -c10 -d30 http://cafe.example.com/coffee -H "version: v2" --latency +echo -e '```' +echo -e "\n## Test3: Running coffee query based routing\n" +echo -e '```console' +wrk -t2 -c10 -d30 http://cafe.example.com/coffee?TEST=v2 --latency +echo -e '```' +echo -e "\n## Test4: Running tea GET method based routing\n" +echo -e '```console' +wrk -t2 -c10 -d30 http://cafe.example.com/tea --latency +echo -e '```' +echo -e "\n## Test5: Running tea POST method based routing\n" +echo -e '```console' +wrk -t2 -c10 -d30 http://cafe.example.com/tea -s ${SCRIPT_DIR}/post.lua --latency +echo -e '```' diff --git a/tests/dataplane-performance/setup.md b/tests/dataplane-performance/setup.md new file mode 100644 index 0000000000..979f007046 --- /dev/null +++ b/tests/dataplane-performance/setup.md @@ -0,0 +1,80 @@ +# Dataplane Performance Testing + +## Goals + +- To capture the average latency of requests being proxied through NGINX using a variety of routing rules, so that we + can see if different routing rules provide different results, and so that we can know if any future work has an impact + on data plane performance. +- A route is created and tested for each routing method below: + - path based routing + - header based routing + - query param based routing + - method based routing + +## Test Environment + +- A Kubernetes cluster with 3 nodes on GKE + - Node: e2-medium (2 vCPU, 4GB memory) +- Tester VM on Google Cloud: + - Instance Type: e2-medium (2 vCPU, 4GB memory) + - Configuration: + - Debian + - Install packages: wrk + - Location - same zone as the Kubernetes cluster. + +## Setup + +1. Create cloud cluster +2. Deploy CRDs: + + ```bash + kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.8.1/standard-install.yaml + ``` + +3. Deploy NGF from edge using Helm install, and expose using an internal LoadBalancer service: + + ```console + helm install my-release oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --version 0.0.0-edge \ + --set service.annotations.'networking\.gke\.io\/load-balancer-type'="Internal" --create-namespace \ + --wait -n nginx-gateway + ``` + +## Tests + +1. First create the test resources. The manifests provided will deploy a single application and service, and two + HTTPRoutes with matches for path based, header based, query based, and method based routing. + + ```console + kubectl apply -f manifests/gateway.yaml + kubectl apply -f manifests/coffee.yaml + kubectl apply -f manifests/cafe-routes.yaml + ``` + +2. Get the external IP of the nginx-gateway service and add an entry in /etc/hosts for ` cafe.example.com`. + + ```console + kubectl get service my-release-nginx-gateway-fabric \ + --namespace nginx-gateway \ + --output jsonpath='{.status.loadBalancer.ingress[0].ip}' + ``` + +3. Copy the scripts in the `scripts/` directory to the test VM. Run tests using `wrk` (change the file output name to + version under test). The tests use `wrk` to send requests to coffee application using each of the configured routing + rules, capturing the number of requests and the average latency. + + ```console + bash wrk-latency.sh > 1.0.0.md + ``` + +4. Analyse the results and check for anomolies. Copy the results file from the test VM into the `results/` directory. + Append any findings or observations to the generated results document. Add test environment information to the + generated document. + +5. Cleanup the deployed resources + + ```console + kubectl delete -f manifests/cafe-routes.yaml + kubectl delete -f manifests/coffee.yaml + kubectl delete -f manifests/gateway.yaml + helm uninstall my-release -n nginx-gateway + ```