Explore reuse of GIE reference implementation to support proposed filter/scorer paradigm

[elevran]

PoC inclusion of the router's Filter and Scorer interfaces and flows in the GIE reference implementation.
The goal is to validate compatibility and "invasiveness" to inform upstream discussions and decisions.

[elevran]

Copying feedback from Shane:

+1, and if there are some specific blockers with the GIE that we could enumerate here that would be helpful so we can stay ahead of these and try to work on improvements in upstream as quickly as possible.

[elevran]

these are the tasks in terms of current PoC development:

• bring up local dev env with existing GIE reference implementation
• modify above to support locally build GIE reference implementation image (make sure this is documented/repeatable, ... References fix: basic container image builds for linux #10 - please close that one in the commit as well 🙏)
• introduce filter and scorer interfaces and hook into request/response processing (goal: minimize continuous merge effort from upstream GIE)
• port over 1-2 filters / scorers
• extension to support flexible attribute collection (e.g., not just metrics, other attributes needed for custom filters/scorers, etc.)

[elevran]

Additional discussion from inference-router issue:

@nirrozenbaum

for PoC development I'm not sure we need flexible attribute collection.
that's an important point in general, but I don't this we need it for the PoC.
PoC will anyway demonstrate specific filter/scorer that relies on specific attribute.

@elevran

Agree. Hence it was made last on the list. We can push it out completely.

@nirrozenbaum

+1 for pushing it out at this point.
timeline is tight and it's preferable to focus on the tasks that are a must.

@nilig

@mayabar created a fork of GIE adding scorers that prioritize filtered pods, + introduced notion of session
that is :

• introduced scorer interfaces and hook into request/response processing
• ported session scorer
  we are working on bring up local dev env with existing GIE reference implementation and test. Should be ready by Monday.
  Once done we will continue with the porting.
  kubernetes-sigs/gateway-api-inference-extension@main...mayabar:gateway-api-inference-> extension:main

[elevran]

Done in llm-d-inference-scheduler