feat: Introduce pluggable intra-flow dispatch policy framework #1139

LukeAVanDrie · 2025-07-11T00:01:45Z

This PR is part of a stack.

PR feat: Introduce pluggable queue framework #1138: Pluggable queues
This PR: Pluggable intra flow dispatch policies

Incremental diff against #1138: LukeAVanDrie/gateway-api-inference-extension@feat/flow-control-framework-plugins-queue...feat/flow-control-frameowrk-plugins-intra-flow-dispatch

This work tracks #674

This PR introduces the second major component of the new pluggable flow control system: the Intra-Flow Dispatch Policy framework.

While the previous PR introduced the SafeQueue framework for managing the storage of requests, this PR introduces the IntraFlowDispatchPolicy framework for managing the logic of request selection. It is responsible for temporal scheduling: determining the order in which requests are selected for dispatch from within a single flow's queue.

Key Contributions

IntraFlowDispatchPolicy Interface: Defines a clear contract for implementing tactical dispatch logic (e.g., FCFS, SLO-aware).
Queue Compatibility: Policies now declare their required queue capabilities (e.g., CapabilityFIFO), and the framework will ensure they are paired with a compatible SafeQueue implementation.
Plugin Factory: A registration mechanism (dispatch.MustRegisterPolicy) allows new policies to be easily added and instantiated by name.
Conformance Test Suite: A new test suite ensures that all registered policy plugins adhere to the core API contract, simplifying validation for future contributions.
FCFS Reference Implementation: Includes a simple, robust "First-Come, First-Served" policy as the first concrete implementation and a reference for future plugins.

Architectural Impact

This change establishes the first tier of our two-tier policy design:

Intra-Flow Policy (This PR): Makes tactical decisions within a single flow's queue.
Inter-Flow Policy (Future Work): Will make strategic decisions about fairness between different flows.

By decoupling the dispatch logic from the data structure, we can now develop and experiment with sophisticated scheduling algorithms without modifying the core flow control engine.

netlify · 2025-07-11T00:01:51Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`a4b94d3`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/687684ea7dfa550008731d20
😎 Deploy Preview	https://deploy-preview-1139--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-07-11T00:01:55Z

Hi @LukeAVanDrie. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

LukeAVanDrie · 2025-07-11T00:02:47Z

/assign @kfswain

/assign @ahg-g

ahg-g · 2025-07-11T16:40:00Z

/ok-to-test

kfswain · 2025-07-14T15:56:50Z

For other viewers: I notice a lot of overlap of files between this & #1138

Focusing on the .../intra-flow/ files

kfswain

Small comment! I do think we should put a hold on this until the subset PR merges

kfswain · 2025-07-14T16:11:57Z

pkg/epp/flowcontrol/framework/plugins/policies/intraflow/dispatch/fcfs/fcfs.go

+}
+
+// fcfs (First-Come, First-Served) implements the `framework.IntraFlowDispatchPolicy` interface.
+type fcfs struct {


Do fcfs & listqueue have a high amount of functional overlap?

Motivation: Intraflow feature completeness is not a primary objective for initial implementation, so wondering if we can reuse code for now until we reach the point that there is functional drift

ListQueue abstracts the request storage.

FCFS abstracts the the request ordering.

We could absorb ListQueue into FCFS (or vice versa) but that will establish patterns that will be messy to undo later when we want to add support for intra-flow dispatch policies. The current design adds minimal boilerplate but sets us up much better for future changes.

kfswain · 2025-07-14T16:16:34Z

/hold

As this PR is a superset of #1138 holding until that one merges

LukeAVanDrie · 2025-07-14T17:08:18Z

For other viewers: I notice a lot of overlap of files between this & #1138

Focusing on the .../intra-flow/ files

@kfswain This is a stacked PR. I put a link in the PR description to see the diff between #1139 and #1138. There is no official support for chained PRs on Github to my knowledge.

This commit introduces the `IntraFlowDispatchPolicy` framework, the second major component of the new pluggable flow control system. This framework decouples the logic for selecting a request from within a single flow's queue (temporal scheduling) from the underlying queue data structure. Key components include: - `framework.IntraFlowDispatchPolicy`: The core interface that defines the contract for selecting an item from a flow's queue. - `framework.FlowQueueAccessor`: A read-only interface that provides policies with safe access to queue state. - `RequiredQueueCapabilities`: A mechanism for policies to declare their queue requirements (e.g., FIFO, priority-ordered), which are validated by the registry. - A factory and registration system for discovering and instantiating policy plugins by name. - A comprehensive conformance test suite to validate the contract for all policy plugins. - A foundational `FCFS` (First-Come, First-Served) policy as the first reference implementation. This work builds directly on the `SafeQueue` framework, enabling the development of sophisticated, policy-driven request prioritization and scheduling.

kfswain · 2025-07-15T20:29:26Z

/lgtm
/approve
/unhold

k8s-ci-robot · 2025-07-15T20:29:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kfswain, LukeAVanDrie

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kfswain]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This commit introduces the `IntraFlowDispatchPolicy` framework, the second major component of the new pluggable flow control system. This framework decouples the logic for selecting a request from within a single flow's queue (temporal scheduling) from the underlying queue data structure. Key components include: - `framework.IntraFlowDispatchPolicy`: The core interface that defines the contract for selecting an item from a flow's queue. - `framework.FlowQueueAccessor`: A read-only interface that provides policies with safe access to queue state. - `RequiredQueueCapabilities`: A mechanism for policies to declare their queue requirements (e.g., FIFO, priority-ordered), which are validated by the registry. - A factory and registration system for discovering and instantiating policy plugins by name. - A comprehensive conformance test suite to validate the contract for all policy plugins. - A foundational `FCFS` (First-Come, First-Served) policy as the first reference implementation. This work builds directly on the `SafeQueue` framework, enabling the development of sophisticated, policy-driven request prioritization and scheduling.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 11, 2025

k8s-ci-robot requested review from liu-cong and nirrozenbaum July 11, 2025 00:01

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 11, 2025

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jul 11, 2025

k8s-ci-robot assigned ahg-g and kfswain Jul 11, 2025

LukeAVanDrie changed the title ~~Feat/flow control frameowrk plugins intra flow dispatch~~ Feat/flow control framework plugins intra flow dispatch Jul 11, 2025

LukeAVanDrie changed the title ~~Feat/flow control framework plugins intra flow dispatch~~ feat: Introduce pluggable intra-flow dispatch policy framework Jul 11, 2025

LukeAVanDrie force-pushed the feat/flow-control-frameowrk-plugins-intra-flow-dispatch branch from 1a35b17 to 7c25ce0 Compare July 11, 2025 00:16

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 11, 2025

kfswain reviewed Jul 14, 2025

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 14, 2025

LukeAVanDrie force-pushed the feat/flow-control-frameowrk-plugins-intra-flow-dispatch branch from 7c25ce0 to 5740f8e Compare July 15, 2025 16:12

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 15, 2025

LukeAVanDrie force-pushed the feat/flow-control-frameowrk-plugins-intra-flow-dispatch branch from 5740f8e to a4b94d3 Compare July 15, 2025 16:42

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Jul 15, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 15, 2025

k8s-ci-robot merged commit 4ffb5f6 into kubernetes-sigs:main Jul 15, 2025
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Introduce pluggable intra-flow dispatch policy framework #1139

feat: Introduce pluggable intra-flow dispatch policy framework #1139

Uh oh!

LukeAVanDrie commented Jul 11, 2025 •

edited

Loading

Uh oh!

netlify bot commented Jul 11, 2025 •

edited

Loading

Uh oh!

k8s-ci-robot commented Jul 11, 2025

Uh oh!

LukeAVanDrie commented Jul 11, 2025

Uh oh!

ahg-g commented Jul 11, 2025

Uh oh!

kfswain commented Jul 14, 2025

Uh oh!

kfswain left a comment

Uh oh!

kfswain Jul 14, 2025

Uh oh!

LukeAVanDrie Jul 14, 2025

Uh oh!

kfswain commented Jul 14, 2025

Uh oh!

LukeAVanDrie commented Jul 14, 2025

Uh oh!

kfswain commented Jul 15, 2025

Uh oh!

k8s-ci-robot commented Jul 15, 2025

Uh oh!

Uh oh!

Uh oh!

feat: Introduce pluggable intra-flow dispatch policy framework #1139

feat: Introduce pluggable intra-flow dispatch policy framework #1139

Uh oh!

Conversation

LukeAVanDrie commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Contributions

Architectural Impact

Uh oh!

netlify bot commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Jul 11, 2025

Uh oh!

LukeAVanDrie commented Jul 11, 2025

Uh oh!

ahg-g commented Jul 11, 2025

Uh oh!

kfswain commented Jul 14, 2025

Uh oh!

kfswain left a comment

Choose a reason for hiding this comment

Uh oh!

kfswain Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

LukeAVanDrie Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

kfswain commented Jul 14, 2025

Uh oh!

LukeAVanDrie commented Jul 14, 2025

Uh oh!

kfswain commented Jul 15, 2025

Uh oh!

k8s-ci-robot commented Jul 15, 2025

Uh oh!

Uh oh!

Uh oh!

LukeAVanDrie commented Jul 11, 2025 •

edited

Loading

netlify bot commented Jul 11, 2025 •

edited

Loading