Migrate the llm-d-inference-scheduler's configuration to the new text based configuration #214

shmuelk · 2025-07-01T14:55:48Z

This PR:

Migrates the llm-d-inference-scheduler to use the new GIE text based configuration capabilities.
Removes the code that created the configuration based on environment variables.
Provides sample configurations based on the old development configurations.
Documents how to configure the system using the new text based configuration capabilities

This PR completes issue #201

Signed-off-by: Shmuel Kallner <[email protected]>

…profile names Signed-off-by: Shmuel Kallner <[email protected]>

…ode profile name Signed-off-by: Shmuel Kallner <[email protected]>

Signed-off-by: Shmuel Kallner <[email protected]>

ahg-g · 2025-07-01T22:11:45Z

deploy/config/epp-config.yaml

+    hashBlockSize: 5
+    maxPrefixBlocksToMatch: 256
+    lruCapacityPerServer: 31250
+- name: decodeFilter


ditto, remove the name?

ahg-g · 2025-07-01T22:12:44Z

deploy/config/epp-config.yaml

+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: prefixScorer


remove the name and rely on the default value just like the other plugins at the end of this config?

ahg-g · 2025-07-02T15:39:47Z

deploy/components/inference-gateway/deployments.yaml

+          mountPath: /etc/epp
+      volumes:
+      - name: epp-config
+        configMap:


How about we place common canned configurations in the container image just so we don't have to create a configmap?

As these are for development scripts, I think it's more important to make it easy to use other configurations as the developer desires.

ahg-g · 2025-07-02T15:40:39Z

deploy/config/epp-kvcache-load-config.yaml

+kind: EndpointPickerConfig
+plugins:
+- type: single-profile
+- name: decodeFilter


ditto, this looks redundant

ahg-g · 2025-07-02T16:01:24Z

docs/architecture.md

+```
+The fields in a plugin entry are:
+- *name* which is optional, provides a name by which the plugin instance can be referenced. If this field is
+omitted, the plugin's type wil be used as its name.<br>


Suggested change

omitted, the plugin's type wil be used as its name.<br>

omitted, the plugin's type will be used as its name.<br>

ahg-g · 2025-07-02T16:05:51Z

docs/architecture.md

+- name: prefixScorer
+  type: prefix-cache


Should we converge on one naming convention, I like using pascal case (e.g., PrefixScorer).

Also, I think the cases where a name needs to be set will be limited, so I recommend to not set it unless required to not give the impression that this is required (which looks strange and redundant at first glance).

Created a discussion issue in GIE: kubernetes-sigs/gateway-api-inference-extension#1086

While there should be a naming convention for plugin types, I don't think that applies as much for plugin names, when used. In particular I made the plugin names, which have been since removed, camel case to show that they are different from the plugin types and the defaulted names.

ahg-g · 2025-07-02T16:06:27Z

pkg/plugins/profile/pd_profile_handler.go

-	decode  = "decode"
-	prefill = "prefill"
+	defaultDecodeProfile   = "decode"
+	defaultPßrefillProfile = "prefill"


Signed-off-by: Shmuel Kallner <[email protected]>

nirrozenbaum · 2025-07-03T10:08:27Z

docs/architecture.md

+specified in-line as a parameter. The configuration defines the set of plugins to be instantiated along with their parameters. Each plugin is also given a name, enabling the same plugin type to be instantiated
+multiple times, if needed. Also defined is a set of SchedulingProfiles, which determine the set of
+plugins to be used when scheduling a request. The set of plugins instantiated must also include a
+Profile Handler, which determines which SchedulingProfiles will be used for a particular request.


++ and how their results will be processed.

nirrozenbaum · 2025-07-03T10:10:50Z

docs/architecture.md

+field is omitted, the plugin's type will be used as its name.<br>
+- *type* specifies the type of the plugin to be instantiated.<br>


do we need <br>?

It doesn't format without all of the <br> in there

Using unordered lists or blank lines make the sections much longer and harder to read

have you tried double space at the end of the line?
I think this should give you the same behavior, just without html tags.

I don't like trailing spaces, which are unseen and easily get lost

nirrozenbaum

overall PR looks very good.
I left few nits, non blocking.

verifying - was this tested e2e?

/lgtm
/approve

shmuelk · 2025-07-03T11:03:30Z

Which end to end tests?

I have used make env-dev-kind to run the entire setup with P/D enabled and with P/D not enabled and the system seemed to do the right thing...

Signed-off-by: Shmuel Kallner <[email protected]>

docs/architecture.md

Signed-off-by: Shmuel Kallner <[email protected]>

Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com> Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>

shmuelk mentioned this pull request Jul 1, 2025

Fixed some typos #212

Merged

nirrozenbaum added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 1, 2025

Gregory-Pereira mentioned this pull request Jul 1, 2025

Include Sim in 0.2 release llm-d/llm-d-deployer#346

Open

shmuelk added 8 commits July 2, 2025 17:33

Updated dependencies

65f25e8

Signed-off-by: Shmuel Kallner <[email protected]>

Configure PdProfileHandler with fields it needs and remove hard code …

9a99ba0

…profile names Signed-off-by: Shmuel Kallner <[email protected]>

Configure PrefillHeaderHandler with fields it needs and remove hard c…

f75e21d

…ode profile name Signed-off-by: Shmuel Kallner <[email protected]>

Removed code that configures the system

31d3e29

Signed-off-by: Shmuel Kallner <[email protected]>

Updated test to use new configuration capabilities

e1998d4

Signed-off-by: Shmuel Kallner <[email protected]>

Sample YAML configuration files

9052afc

Signed-off-by: Shmuel Kallner <[email protected]>

Updated development scripts to create ConfigMap of configuration

d3216b6

Signed-off-by: Shmuel Kallner <[email protected]>

Documentation updates

b4ca78a

Signed-off-by: Shmuel Kallner <[email protected]>

shmuelk force-pushed the no-code-config branch from 416e5de to b4ca78a Compare July 2, 2025 14:38

shmuelk changed the title ~~[WIP] Migrate the llm-d-inference-scheduler's configuration to the new text based configuration~~ Migrate the llm-d-inference-scheduler's configuration to the new text based configuration Jul 2, 2025

shmuelk removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 2, 2025

shmuelk requested review from elevran, mayabar and nirrozenbaum July 2, 2025 14:40

ahg-g reviewed Jul 2, 2025

View reviewed changes

elevran mentioned this pull request Jul 2, 2025

Feature: allow customization of plug-in chains and configurations #9

Closed

3 tasks

shmuelk added 2 commits July 3, 2025 10:03

Updates from review

124d4e6

Signed-off-by: Shmuel Kallner <[email protected]>

Reformatted text a bit

b8ec020

Signed-off-by: Shmuel Kallner <[email protected]>

shmuelk mentioned this pull request Jul 3, 2025

remove dependency on datastore + updated TypedName of plugins #215

Merged

nirrozenbaum reviewed Jul 3, 2025

View reviewed changes

nirrozenbaum previously approved these changes Jul 3, 2025

View reviewed changes

Update from review comment

29919c0

Signed-off-by: Shmuel Kallner <[email protected]>

shmuelk dismissed nirrozenbaum’s stale review via 29919c0 July 3, 2025 11:05

mayabar suggested changes Jul 3, 2025

View reviewed changes

Updates from review comments

4a96923

Signed-off-by: Shmuel Kallner <[email protected]>

Improved documentation of ByLabelSelector filter

ead71ac

Signed-off-by: Shmuel Kallner <[email protected]>

nirrozenbaum approved these changes Jul 3, 2025

View reviewed changes

mayabar approved these changes Jul 3, 2025

View reviewed changes

shmuelk merged commit 31aff9d into llm-d:main Jul 3, 2025
2 checks passed

shmuelk deleted the no-code-config branch July 3, 2025 13:07

nirrozenbaum mentioned this pull request Jul 12, 2025

Configure the llm-d-inference-scheduler using the new GIE configuration support #201

Closed

	omitted, the plugin's type wil be used as its name.<br>
	omitted, the plugin's type will be used as its name.<br>

		field is omitted, the plugin's type will be used as its name.<br>
		- type specifies the type of the plugin to be instantiated.<br>

Migrate the llm-d-inference-scheduler's configuration to the new text based configuration #214

Migrate the llm-d-inference-scheduler's configuration to the new text based configuration #214

Uh oh!

Conversation

shmuelk commented Jul 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nirrozenbaum Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nirrozenbaum left a comment

Choose a reason for hiding this comment

Uh oh!

shmuelk commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nirrozenbaum Jul 3, 2025 •

edited

Loading