Provide Flexability in InferencePool Assignment

[danehans]

Currently, EPP will only reconcile an InferencePool that matches its configured poolName CLI flag. A more flexible approach should be considered since changing InferencePools requires a restart. One approach to consider is using labels, where EPP can use a predicate to filter reconciling all InferencePools in its namespace to only one that matches its configured poolLabel matcher. EPP can follow Gateway API guidelines for conflict resolution.

[danehans]

@ahg-g @kfswain @robscott PTAL and provide your thoughts when you have a moment.

[ahg-g]

I am supportive, would you like to make a concrete proposal? We need to resolve a few things:

1. If we are going with a label, define a label key
2. How to ensure that this aligns with the configuration api that defines a service reference? Should the EPP look at that instead? or spec that the EPP service reference and the label must be aligned, and if not the behavior is unknown?
3. Since our current implementation of EPP assumes a single inferencePool, we need to also define a predictable behavior as to what exact pool the EPP will serve until we decide whether or not we want EPP to support multiple pools.

[danehans]

Let's see if others in the community are interested in this feature before proceeding with a design doc.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned