Skip to content

Conversation

@salonichf5
Copy link
Contributor

@salonichf5 salonichf5 commented Nov 18, 2025

Proposed changes

Write a clear and concise description that helps reviewers understand the purpose and impact of your changes. Use the
following format:

Problem: Users want to be able to specify session persistence for their upstreams.

Solution: Add support for session persistence using sticky cookie directives which is only available for NGINX Plus users

Testing: Manual testing

Errors

Spec validation

The HTTPRoute "coffee" is invalid: 
* spec.rules[0].sessionPersistence.sessionName: Too long: may not be more than 128 bytes
* <nil>: Invalid value: null: some validation rules were not checked because the object was invalid; correct the existing errors to complete validation


    sessionPersistence:
      sessionName: "coffee-shop-session-with-very-long-name-for-testing-session-affinity-functionality-in-kubernetes-gateway-api-implementation-v1-hello"
The HTTPRoute "tea" is invalid: spec.rules[0].sessionPersistence.absoluteTimeout: Invalid value: "99999d": spec.rules[0].sessionPersistence.absoluteTimeout in body should match '^([0-9]{1,5}(h|m|s|ms)){1,4}$'

The HTTPRoute "tea" is invalid: spec.rules[0].sessionPersistence.absoluteTimeout: Invalid value: "999d": spec.rules[0].sessionPersistence.absoluteTimeout in body should match '^([0-9]{1,5}(h|m|s|ms)){1,4}$'
httproute.gateway.networking.k8s.io/tea configured
The HTTPRoute "coffee" is invalid: spec.rules[0].sessionPersistence: Invalid value: "object": AbsoluteTimeout must be specified when cookie lifetimeType is Permanent

    sessionPersistence:
      sessionName: "coffee-cookie"
      cookieConfig:
        lifetimeType: Permanent
      Message:               The following unsupported parameters were ignored: [spec.rules[0].sessionPersistence.type: Unsupported value: "Header": supported values: "Cookie", spec.rules[0].sessionPersistence: Invalid value: "spec.rules[0].sessionPersistence": session persistence is ignored because there are errors in the configuration]

    sessionPersistence:
      sessionName: "coffee-cookie"
      type: Header
      absoluteTimeout: 600s
      cookieConfig:
        lifetimeType: Permanent
httproute.gateway.networking.k8s.io/tea configured
The HTTPRoute "coffee" is invalid: 
* spec.rules[0].sessionPersistence.type: Unsupported value: "cookie": supported values: "Cookie", "Header"
* <nil>: Invalid value: null: some validation rules were not checked because the object was invalid; correct the existing errors to complete validation

    sessionPersistence:
      sessionName: "coffee-cookie"
      type: cookie
      absoluteTimeout: 600s
      cookieConfig:
        lifetimeType: Permanent
      Last Transition Time:  2025-11-18T01:28:53Z
      Message:               The following unsupported parameters were ignored: [spec.rules[0].sessionPersistence.idleTimeout: Invalid value: "300s": idleTimeout is not supported, spec.rules[0].sessionPersistence: Invalid value: "spec.rules[0].sessionPersistence": session persistence is ignored because there are errors in the configuration]

    sessionPersistence:
      sessionName: "coffee-cookie"
      type: Cookie
      idleTimeout: 300s
      absoluteTimeout: 600s
      cookieConfig:
        lifetimeType: Permanent

NGF Errors

non-plus behaviour

      Last Transition Time:  2025-11-24T23:29:42Z
      Message:               All references are resolved
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2025-11-24T23:29:42Z
      Message:               The following unsupported parameters were ignored: spec.rules[0].sessionPersistence: Forbidden: SessionPersistence is only supported in NGINX Plus. This configuration will be ignored.
      Observed Generation:   1
      Reason:                UnsupportedField
      Status:                True
      Type:                  Accepted
    Controller Name:         gateway.nginx.org/nginx-gateway-controller
    Parent Ref:

non-experimental and non-plus behaviour

Parents:
    Conditions:
      Last Transition Time:  2025-11-24T23:32:30Z
      Message:               All references are resolved
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2025-11-24T23:32:30Z
      Message:               The following unsupported parameters were ignored: [spec.rules[0].sessionPersistence: Forbidden: SessionPersistence is only supported in NGINX Plus. This configuration will be ignored., spec.rules[0].sessionPersistence: Forbidden: SessionPersistence is only supported in experimental mode.]
      Observed Generation:   1
      Reason:                UnsupportedField
      Status:                True
      Type:                  Accepted
    Controller Name:         gateway.nginx.org/nginx-gateway-controller
    Parent Ref:
      Group:         gateway.networking.k8s.io
      Kind:          Gateway
      Name:          gateway
      Namespace:     default
      Section Name:  http
Events:              <none>

Configuration

HTTPRoutes

Case 1: path and expires specified

    sessionPersistence:
      sessionName: "coffee-cookie"
      type: Cookie
      absoluteTimeout: 24h
      cookieConfig:
        lifetimeType: Permanent

upstream default_coffee-v1-svc_80_coffee_default_0 {
    random two least_conn;
    zone default_coffee-v1-svc_80_coffee_default_0 1m;
    sticky cookie coffee-cookie expires=1440m path=/coffee;

    state /var/lib/nginx/state/default_coffee-v1-svc_80_coffee_default_0.conf;
}

Case 2 : no expiry, only path specified

    sessionPersistence:
      sessionName: "coffee-cookie"
      type: Cookie
      absoluteTimeout: 24h
      cookieConfig:
        lifetimeType: Session. —> no expiry



upstream default_coffee-v1-svc_80_coffee_default_0 {
    random two least_conn;
    zone default_coffee-v1-svc_80_coffee_default_0 1m;
    sticky cookie coffee-cookie path=/coffee;

    state /var/lib/nginx/state/default_coffee-v1-svc_80_coffee_default_0.conf;
  

Regular expression with other path matches (no path set)

rules:
  - matches:
    - path:
        type: PathPrefix
        value: /coffee
    - path:
        type: Exact
        value: /coffee/espresso
    - path:
        type: RegularExpression
        value: /coffee/[a-zA-Z0-9]+
    backendRefs:
    - name: coffee
      port: 80
    sessionPersistence:
      sessionName: "coffee-cookie"
      type: Cookie
      absoluteTimeout: 24h
      cookieConfig:
        lifetimeType: Permanent



upstream default_coffee-v1-svc_80_coffee_default_0 {
    random two least_conn;
    zone default_coffee-v1-svc_80_coffee_default_0 1m;
    sticky cookie coffee-cookie; --.> no path

    state /var/lib/nginx/state/default_coffee-v1-svc_80_coffee_default_0.conf;

Multiple matches with common prefix

 rules:
  - matches:
    - path:
        type: PathPrefix
        value: /coffee
    - path:
        type: Exact
        value: /coffee/espresso
    - path:
        type: PathPrefix
        value: /coffee/tea
    - path:
        type: Exact
        value: /coffee/latte
    backendRefs:
    - name: coffee
      port: 80
    sessionPersistence:
      sessionName: "coffee-cookie"
      type: Cookie
      absoluteTimeout: 24h
      cookieConfig:
        lifetimeType: Permanent



upstream default_coffee-v1-svc_80_coffee_default_0 {
    random two least_conn;
    zone default_coffee-v1-svc_80_coffee_default_0 1m;
    sticky cookie coffee-cookie path=/coffee;  --> common path
 
    state /var/lib/nginx/state/default_coffee-v1-svc_80_coffee_default_0.conf;

multiple matches with no common prefix

  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /1
    - path:
        type: Exact
        value: /2
    - path:
        type: PathPrefix
        value: /3
    - path:
        type: Exact
        value: /4
    backendRefs:
    - name: coffee
      port: 80
    sessionPersistence:
      sessionName: "coffee-cookie"
      type: Cookie
      absoluteTimeout: 24h
      cookieConfig:
        lifetimeType: Permanent


upstream default_coffee-v1-svc_80_coffee_default_0 {
    random two least_conn;
    zone default_coffee-v1-svc_80_coffee_default_0 1m;
    sticky cookie coffee-cookie; --> no path

    state /var/lib/nginx/state/default_coffee-v1-svc_80_coffee_default_0.conf;

GRPCRoutes (path is always empty for GRPCRoutes

Hostname matching

Hostname matching

  rules:
  - backendRefs:
    - name: grpc-infra-backend-v2
      port: 8080
    sessionPersistence:
      sessionName: "coffee-cookie"
      type: Cookie
      absoluteTimeout: 24h
      cookieConfig:
        lifetimeType: Permanent

upstream default_grpc-infra-backend-v1_8080 {
    random two least_conn;
    zone default_grpc-infra-backend-v1_8080 1m;

    state /var/lib/nginx/state/default_grpc-infra-backend-v1_8080.conf;

}

upstream default_grpc-infra-backend-v2_8080_backend-v2_default_0 {
    random two least_conn;
    zone default_grpc-infra-backend-v2_8080_backend-v2_default_0 1m;
    sticky cookie coffee-cookie expires=1440m;

    state /var/lib/nginx/state/default_grpc-infra-backend-v2_8080_backend-v2_default_0.conf;

Exact Method matching

  rules:
  - matches:
    - method:
        service: helloworld.Greeter
        method: SayHello
    backendRefs:
    - name: grpc-infra-backend-v1
      port: 8080
    sessionPersistence:
      sessionName: "coffee-cookie"
      type: Cookie
      absoluteTimeout: 24h
      cookieConfig:
        lifetimeType: Session


upstream default_grpc-infra-backend-v1_8080_exact-matching_default_0 {
    random two least_conn;
    zone default_grpc-infra-backend-v1_8080_exact-matching_default_0 1m;
    sticky cookie coffee-cookie;

    state /var/lib/nginx/state/default_grpc-infra-backend-v1_8080_exact-matching_default_0.conf;

Traffic Splitting with HTTPRoutes

  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /coffee-v1
    backendRefs:
    - name: coffee-v1-svc
      port: 80
      weight: 80
    - name: coffee-v2-svc
      port: 80
      weight: 20
    sessionPersistence:
      sessionName: "coffee-cookie"
      type: Cookie
      absoluteTimeout: 10m
      cookieConfig:
        lifetimeType: Permanent

  - matches:
    - path:
        type: PathPrefix
        value: /coffee-diff-backend
    backendRefs:
    - name: coffee-v2-svc
      port: 80

  - matches:
    - path:
        type: PathPrefix
        value: /coffee-v2
    backendRefs:
    - name: coffee-v1-svc
      port: 80
    sessionPersistence:
      sessionName: "coffee-cookie-v2"
      type: Cookie
      absoluteTimeout: 24h
      cookieConfig:
        lifetimeType: Permanent

Upstreams generated

upstream default_coffee-v1-svc_80_coffee_default_0 {
    random two least_conn;
    zone default_coffee-v1-svc_80_coffee_default_0 1m;
    sticky cookie coffee-cookie expires=600s path=/coffee-v1;

    state /var/lib/nginx/state/default_coffee-v1-svc_80_coffee_default_0.conf;

}

upstream default_coffee-v1-svc_80_coffee_default_2 {
    random two least_conn;
    zone default_coffee-v1-svc_80_coffee_default_2 1m;
    sticky cookie coffee-cookie-v2 expires=1440m path=/coffee-v2;

    state /var/lib/nginx/state/default_coffee-v1-svc_80_coffee_default_2.conf;

}

upstream default_coffee-v2-svc_80 {
    random two least_conn;
    zone default_coffee-v2-svc_80 1m;

    state /var/lib/nginx/state/default_coffee-v2-svc_80.conf;

}

upstream default_coffee-v2-svc_80_coffee_default_0 {
    random two least_conn;
    zone default_coffee-v2-svc_80_coffee_default_0 1m;
    sticky cookie coffee-cookie expires=600s path=/coffee-v1;

    state /var/lib/nginx/state/default_coffee-v2-svc_80_coffee_default_0.conf;

}

split_clients $request_id $group_default__coffee_rule0 {
    80.00% default_coffee-v1-svc_80_coffee_default_0;
    20.00% default_coffee-v2-svc_80_coffee_default_0;
}

location ^~ /coffee-v1/ {
  proxy_pass http://$group_default__coffee_rule0$request_uri;
}

Testing Session Persistence

Testing is done using a script by grabbing the cookie session id from the first request and using it in curl request to ensure each request goes to the same backend

HTTPRoutes (coffee -> sticky, tea -> regular)


upstream default_coffee_80 {
    random two least_conn;
    zone default_coffee_80 1m;
    sticky cookie coffee-cookie expires=600s path=/coffee;

    state /var/lib/nginx/state/default_coffee_80.conf; --> 3 endpoints
}

upstream default_tea_80 {
    random two least_conn;
    zone default_tea_80 1m;

    state /var/lib/nginx/state/default_tea_80.conf; --> 3 endpoints
}
curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/coffee
< Set-Cookie: coffee-cookie=2b13cc98f0224fca47885f32115ef236; expires=Tue, 18-Nov-25 22:17:55 GMT; max-age=600; path=/coffee


Summary (how many times each backend was chosen):
 100 10.244.0.111:8080
curl -v --resolve cafe.example.com:$GW_PORT:$GW_IP http://cafe.example.com:$GW_PORT/tea
Summary (how many times each backend was chosen):
  32 10.244.0.108:8080
  38 10.244.0.112:8080
  30 10.244.0.113:8080

GRPCRoutes (backend v1 --> normal, backend v2 --> sticky)

Each have 3 endpoints


upstream default_grpc-infra-backend-v1_8080 {
    random two least_conn;
    zone default_grpc-infra-backend-v1_8080 1m;

    state /var/lib/nginx/state/default_grpc-infra-backend-v1_8080.conf;

}

upstream default_grpc-infra-backend-v2_8080 {
    random two least_conn;
    zone default_grpc-infra-backend-v2_8080 1m;
    sticky cookie grpc-v2-cookie expires=600s;

    state /var/lib/nginx/state/default_grpc-infra-backend-v2_8080.conf;

}
backend 1
grpcurl -v -plaintext -proto grpc.proto -authority bar.com -d '{"name": "bar server"}' ${GW_IP}:${GW_PORT} helloworld.Greeter/SayHello
Handling connection for 8080

Resolved method descriptor:
// Sends a greeting
rpc SayHello ( .helloworld.HelloRequest ) returns ( .helloworld.HelloReply );

Request metadata to send:
(empty)

Response headers received:
content-type: application/grpc
date: Tue, 18 Nov 2025 22:26:41 GMT
server: nginx

Response contents:
{
  "message": "Hello bar server"
}

Response trailers received:
(empty)
Sent 1 request and received 1 response

backend 2

rpcurl -v -plaintext -proto grpc.proto -authority foo.bar.com -d '{"name": "bar server"}' ${GW_IP}:${GW_PORT} helloworld.Greeter/SayHello
Handling connection for 8080

Resolved method descriptor:
// Sends a greeting
rpc SayHello ( .helloworld.HelloRequest ) returns ( .helloworld.HelloReply );

Request metadata to send:
(empty)

Response headers received:
content-type: application/grpc
date: Tue, 18 Nov 2025 22:27:28 GMT
server: nginx
set-cookie: grpc-v2-cookie=cc8db88effa3e2563d0b5b13054dc7cc; expires=Tue, 18-Nov-25 22:37:28 GMT; max-age=600

Response contents:
{
  "message": "Hello bar server"
}

Response trailers received:
(empty)
Sent 1 request and received 1 response
sa.choudhary@N9939CQ4P0 grpc-routing % 

Note:

  • Errors are not added for API spec rules.
  • For RegularExpression path matches, we leave the cookie Path empty. A regex can match anywhere within the URL path (for example, /coffee/[A-Za-z]+/tea), so deriving a concrete cookie path from it would be misleading and could unintentionally restrict which requests send the cookie.

Please focus on (optional): If you any specific areas where you would like reviewers to focus their attention or provide
specific feedback, add them here.

Closes #4231

Checklist

Before creating a PR, run through this checklist and mark each as complete.

  • I have read the CONTRIBUTING doc
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked that all unit tests pass after adding my changes
  • I have updated necessary documentation
  • I have rebased my branch onto main
  • I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

NONE

@github-actions github-actions bot added the enhancement New feature or request label Nov 18, 2025
@salonichf5 salonichf5 changed the title Add session persistence support for NGINX Plus users using the sticky cookie directive Do not review: Add session persistence support for NGINX Plus users using the sticky cookie directive Nov 19, 2025
Comment on lines +201 to +205
units := []unit{
{"ms", 1},
{"s", 1000},
{"m", 60 * 1000},
{"h", 60 * 60 * 1000},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can switch its order if we want larger units first.

@salonichf5 salonichf5 force-pushed the feat/plus-session-persistence branch from 622cc92 to ded1099 Compare November 19, 2025 14:47
@salonichf5 salonichf5 changed the title Do not review: Add session persistence support for NGINX Plus users using the sticky cookie directive Add session persistence support for NGINX Plus users using the sticky cookie directive Nov 19, 2025
@salonichf5 salonichf5 marked this pull request as ready for review November 19, 2025 14:49
@salonichf5 salonichf5 requested a review from a team as a code owner November 19, 2025 14:49
@sjberman
Copy link
Collaborator

FYI, the release note in the PR description isn't going to be used at all since this is being merged into the feature branch. For the main PR that we merge at the end, we'll want a descriptive release note to discuss the different ways session persistence is supported.

@nginx-bot nginx-bot bot removed the release-notes label Nov 19, 2025
@sjberman
Copy link
Collaborator

See my comment on the other PR, I think we have to rethink this a bit to support a backend being referenced multiple times, to prevent unintended behavior or blocking desired behavior.

@salonichf5 salonichf5 changed the title Add session persistence support for NGINX Plus users using the sticky cookie directive DNR: Add session persistence support for NGINX Plus users using the sticky cookie directive Nov 19, 2025
@salonichf5 salonichf5 force-pushed the feat/session-persistence branch 2 times, most recently from 9863bd1 to 758a284 Compare November 24, 2025 17:50
@salonichf5 salonichf5 force-pushed the feat/plus-session-persistence branch from 1b4dc06 to fc5c7cb Compare November 24, 2025 18:13
@salonichf5 salonichf5 force-pushed the feat/plus-session-persistence branch 2 times, most recently from edd4d9e to fcad961 Compare November 25, 2025 04:53
@salonichf5 salonichf5 changed the title DNR: Add session persistence support for NGINX Plus users using the sticky cookie directive Add session persistence support for NGINX Plus users using the sticky cookie directive Nov 25, 2025
@salonichf5 salonichf5 force-pushed the feat/plus-session-persistence branch from fcad961 to 864ddd8 Compare November 25, 2025 05:00
Name: sp.Name,
Expiry: sp.Expiry,
Path: sp.Path,
SessionType: string(sp.SessionType),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow! this PR is just huge work and so impressing!
I'm just wondering how we get the exact cookie from the
< Set-Cookie: coffee-cookie=2b13cc98f0224fca47885f32115ef236; expires=Tue, 18-Nov-25 22:17:55 GMT; max-age=600; path=/coffee
to fill in
sticky cookie coffee-cookie expires=1440m path=/coffee
Is this mechanism somewhere inside NGINX?
The cookie isn't something automatically created it can be set by user during the very first request and then LB will always check any new request for the same cookie?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great question.

When we configure:

upstream coffee_backend {
    server backend1;
    server backend2;

    sticky cookie coffee-cookie expires=1440m path=/coffee;
}

we’re not copying an existing cookie into NGINX – we’re just telling NGINX: use a sticky session cookie named coffee-cookie and set its attributes (expires, path, etc.)

On the first request from a client that doesn’t have this cookie yet, NGINX: Picks a backend using the normal Load balancing algorithm. Generates an cookie token (for example 2b13cc98f0224fca47885f32115ef236) and stores a mapping value → backend and sends a Set-Cookie header like:

Set-Cookie: coffee-cookie=2b13cc98f0224fca47885f32115ef236; ...; path=/coffee

The browser then includes that cookie on subsequent requests: Cookie: coffee-cookie=2b13cc98f0224fca47885f32115ef236 and NGINX Plus uses the cookie value to route the request back to the same upstream server. You never need to copy the cookie value into the config; it’s generated and managed by NGINX itself.

@salonichf5 salonichf5 requested a review from a team November 25, 2025 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

Status: 🆕 New

Development

Successfully merging this pull request may close these issues.

4 participants