moved main code to runner package under epp/cmd #956

nirrozenbaum · 2025-06-10T17:36:24Z

This PR lays the ground for having an extensible epp main with minimal changes.
the code in this PR moves the code in current main.go to runner package under epp/cmd.
ALMOST NO CODE CHANGES were made except for:

making the run function public (now it's called Run).
added two simple functions to allow configuring the extension points from any main.go, e.g., from llm-d main.go.
This can be done via the Runner.WithSchedulerConfig(...) function or Runner.WithRequestControlConfig(...) which is used to configure the PostResponse plugins.

That's it. NO OTHER CHANGES.

main.go now looks like the following:

func main() {
	if err := runner.NewRunner().Run(); err != nil {
		os.Exit(1)
	}
}

** In this PR there is no intention to make improvements to main package, logging, or any other structuring issue than the one that is mentioned in the intro. This is only a first step out of a series of PRs that will be added to GIE to improve the way we initialize and run EPP in an extensible manner.

cc: @ahg-g @kfswain @elevran

netlify · 2025-06-10T17:36:30Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`658185a`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/684980614cac6800080bd206
😎 Deploy Preview	https://deploy-preview-956--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

nirrozenbaum · 2025-06-10T17:58:21Z

@JeffLuoo are you able to assist?
I'm remember you added the CommitSHA part. this PR is changing cmd/main, and fails to build both locally and in ci/cd.
seems to be related to CommitSHA. do you know what am I missing?

[builder 11/11] RUN go build -ldflags="-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=2162cc40dd02ee85876c675ed9ab003078ed66c0 -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=" -o /epp
#17 0.522 main.go:22:2: no required module provides package sigs.k8s.io/gateway-api-inference-extension/cmd/epp/runner; to add it:
#17 0.522 	go get sigs.k8s.io/gateway-api-inference-extension/cmd/epp/runner
#17 ERROR: process "/bin/sh -c go build -ldflags=\"-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=${COMMIT_SHA} -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=${BUILD_REF}\" -o /epp" did not complete successfully: exit code: 1
------
 > [builder 11/11] RUN go build -ldflags="-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=2162cc40dd02ee85876c675ed9ab003078ed66c0 -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=" -o /epp:
0.522 main.go:22:2: no required module provides package sigs.k8s.io/gateway-api-inference-extension/cmd/epp/runner; to add it:
0.522 	go get sigs.k8s.io/gateway-api-inference-extension/cmd/epp/runner
------
Dockerfile:26
--------------------
  24 |     COPY api ./api
  25 |     WORKDIR /src/cmd
  26 | >>> RUN go build -ldflags="-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=${COMMIT_SHA} -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=${BUILD_REF}" -o /epp
  27 |     
  28 |     ## Multistage deploy
--------------------
ERROR: failed to solve: process "/bin/sh -c go build -ldflags=\"-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=${COMMIT_SHA} -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=${BUILD_REF}\" -o /epp" did not complete successfully: exit code: 1
make: *** [Makefile:184: image-build] Error 1
+ EXIT_VALUE=2
+ set +o xtrace
Cleaning up after docker in docker.

JeffLuoo · 2025-06-10T18:54:22Z

@JeffLuoo are you able to assist? I'm remember you added the CommitSHA part. this PR is changing cmd/main, and fails to build both locally and in ci/cd. seems to be related to CommitSHA. do you know what am I missing?

[builder 11/11] RUN go build -ldflags="-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=2162cc40dd02ee85876c675ed9ab003078ed66c0 -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=" -o /epp
#17 0.522 main.go:22:2: no required module provides package sigs.k8s.io/gateway-api-inference-extension/cmd/epp/runner; to add it:
#17 0.522 	go get sigs.k8s.io/gateway-api-inference-extension/cmd/epp/runner
#17 ERROR: process "/bin/sh -c go build -ldflags=\"-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=${COMMIT_SHA} -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=${BUILD_REF}\" -o /epp" did not complete successfully: exit code: 1
------
 > [builder 11/11] RUN go build -ldflags="-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=2162cc40dd02ee85876c675ed9ab003078ed66c0 -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=" -o /epp:
0.522 main.go:22:2: no required module provides package sigs.k8s.io/gateway-api-inference-extension/cmd/epp/runner; to add it:
0.522 	go get sigs.k8s.io/gateway-api-inference-extension/cmd/epp/runner
------
Dockerfile:26
--------------------
  24 |     COPY api ./api
  25 |     WORKDIR /src/cmd
  26 | >>> RUN go build -ldflags="-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=${COMMIT_SHA} -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=${BUILD_REF}" -o /epp
  27 |     
  28 |     ## Multistage deploy
--------------------
ERROR: failed to solve: process "/bin/sh -c go build -ldflags=\"-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=${COMMIT_SHA} -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=${BUILD_REF}\" -o /epp" did not complete successfully: exit code: 1
make: *** [Makefile:184: image-build] Error 1
+ EXIT_VALUE=2
+ set +o xtrace
Cleaning up after docker in docker.

COMMIT_SHA has default value unknown from https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/Dockerfile#L11. Can you try change the line:

RUN go build -ldflags="-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=${COMMIT_SHA} -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=${BUILD_REF}" -o /epp

to just:

RUN go build -o /epp

to see if it works?

ahg-g · 2025-06-10T21:23:51Z

fyi, https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/951/files will cause a conflict

ahg-g · 2025-06-11T06:30:01Z

fyi, https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/951/files will cause a conflict

I put a hold on 951 since the parameters should move to the config api anyway

Signed-off-by: Nir Rozenbaum <[email protected]>

… layer Signed-off-by: Nir Rozenbaum <[email protected]>

Signed-off-by: Nir Rozenbaum <[email protected]>

nirrozenbaum · 2025-06-11T13:24:54Z

@JeffLuoo ok I think I caught the problem. it was a required line change in the Dockerfile, not related to CommitSHA.
you can see in the files tab the change in the Dockerfile.
your question did help me find it, thanks :)

kfswain · 2025-06-11T15:40:55Z

Have we validated this manually? Hitting CI issues with the build makes me think we need some stronger validation

ahg-g

@shmuelk this will cause a great conflict with your PR, need to figure out which one to get in first, probably this one since it is "simpler"

ahg-g · 2025-06-12T02:29:57Z

cmd/epp/runner/runner.go

+	return r
+}
+
+func (r *Runner) Run() error {


Is this an exact copy of the logic that existed in main.go? if not, can you point out the diff?

yes, exact copy with the only difference of initialization of the scheduler.

nirrozenbaum · 2025-06-12T03:35:27Z

@shmuelk this will cause a great conflict with your PR, need to figure out which one to get in first, probably this one since it is "simpler"

right. I took it into consideration and talked with Shmuel. obviously things may change here, but this simple change will allow us to get rid of all main related code duplication in llm-d.

working iteratively, when his big PR merges we will do the adaptations.

ahg-g · 2025-06-12T03:46:09Z

Are you able to run the e2e test?

ahg-g · 2025-06-12T04:28:48Z

/approve
/lgtm

I will get approve this to unblock the config api PR, it will trigger GKE's internal regression test, but please also try to run the e2e test.

k8s-ci-robot · 2025-06-12T04:28:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, nirrozenbaum

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

nirrozenbaum · 2025-06-12T05:09:37Z

@ahg-g sure, e2e-test pass. missed your question.

nirrozenbaum · 2025-06-12T15:52:49Z

Have we validated this manually? Hitting CI issues with the build makes me think we need some stronger validation

@kfswain sure 👍🏻

* moved main code to runner package under epp/cmd Signed-off-by: Nir Rozenbaum <[email protected]> * added the ability to configure postResponse plugins in requestcontrol layer Signed-off-by: Nir Rozenbaum <[email protected]> * updated dockerfile Signed-off-by: Nir Rozenbaum <[email protected]> --------- Signed-off-by: Nir Rozenbaum <[email protected]>

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 10, 2025

k8s-ci-robot requested review from danehans and robscott June 10, 2025 17:36

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jun 10, 2025

nirrozenbaum force-pushed the runner branch from b8e0a33 to 9662d0c Compare June 10, 2025 18:21

keithmattix mentioned this pull request Jun 10, 2025

bug: istio gateway omits headers #934

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 11, 2025

nirrozenbaum added 2 commits June 11, 2025 15:46

moved main code to runner package under epp/cmd

cd49b1a

Signed-off-by: Nir Rozenbaum <[email protected]>

added the ability to configure postResponse plugins in requestcontrol…

199f57e

… layer Signed-off-by: Nir Rozenbaum <[email protected]>

nirrozenbaum force-pushed the runner branch from f844b9c to 199f57e Compare June 11, 2025 12:50

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 11, 2025

updated dockerfile

658185a

Signed-off-by: Nir Rozenbaum <[email protected]>

ahg-g reviewed Jun 12, 2025

View reviewed changes

k8s-ci-robot assigned ahg-g Jun 12, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 12, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 12, 2025

ahg-g mentioned this pull request Jun 12, 2025

Consider moving internal packages into pkg? #920

Closed

k8s-ci-robot merged commit 1b5fb26 into kubernetes-sigs:main Jun 12, 2025
8 checks passed

nirrozenbaum deleted the runner branch June 12, 2025 05:10

ahg-g mentioned this pull request Jun 12, 2025

Remove duplicate health check code llm-d/llm-d-inference-scheduler#181

Closed

nirrozenbaum mentioned this pull request Jun 17, 2025

Remove internal packages and reuse from GIE when available llm-d/llm-d-inference-scheduler#173

Closed

elevran mentioned this pull request Jul 24, 2025

REQUEST: New membership for elevran kubernetes/org#5729

Closed

11 tasks

moved main code to runner package under epp/cmd #956

moved main code to runner package under epp/cmd #956

Uh oh!

Conversation

nirrozenbaum commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

nirrozenbaum commented Jun 10, 2025

Uh oh!

JeffLuoo commented Jun 10, 2025

Uh oh!

ahg-g commented Jun 10, 2025

Uh oh!

ahg-g commented Jun 11, 2025

Uh oh!

nirrozenbaum commented Jun 11, 2025

Uh oh!

kfswain commented Jun 11, 2025

Uh oh!

ahg-g left a comment

Choose a reason for hiding this comment

Uh oh!

ahg-g Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

nirrozenbaum Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

nirrozenbaum commented Jun 12, 2025

Uh oh!

ahg-g commented Jun 12, 2025

Uh oh!

ahg-g commented Jun 12, 2025

Uh oh!

k8s-ci-robot commented Jun 12, 2025

Uh oh!

Uh oh!

nirrozenbaum commented Jun 12, 2025

Uh oh!

nirrozenbaum commented Jun 12, 2025

Uh oh!

Uh oh!

nirrozenbaum commented Jun 10, 2025 •

edited

Loading

netlify bot commented Jun 10, 2025 •

edited

Loading