vLLM backend for Cortex

[ramonpzg]

At the moment, the process by which a user can build a custom python engine to deploy a model via cortex is not straightforward in code or clear in the docs. The plan is:

• improve the process for building a custom python engine
• remove unnecessary parameters in the model.yml config
• improve the documentation
• add examples of different custom engines to the docs

Goals

• vllm is python lib to run on linux machine
• vllm can run llm

Tasks

Obstacle

• huge effort to maintain this in cortex

Out of scope

• Audio

[gau-nernst]

A "Python engine" actually has a larger scope than the current llama.cpp engine:

• Python engine should be able to run any arbitrary Python programs
• llama.cpp can only run models supported by llama.cpp, and must be in supported format i.e. GGUF.

Therefore, having a Python engine means we have to solve the 2 following problems

1. Dependency management
2. How to expose Python engine

Dependency management

uv is the new cool kid in town. Not only it's reportedly more robust than pip+venv, it's fast and provides many convenience methods.

Two interesting "modes"

• Script mode: single .py file
  • Declare deps on top of the file
  • uv run app.py -> automatically install deps, put in an isolated env, run the app
• Project mode: a folder with pyproject.toml
  • Normal Python project. Declare deps within pyproject.toml
  • Within the project folder uv run main.py -> automatically create a local .venv, install deps, install the Python project itself, and run the app
  • If the Python project exposes a CLI command, it's possible to run the CLI command too.

How to expose Python engine

cortex-server is the core C++ webserver (in Drogon) that serves HTTP requests. We can either

1. Embed Python interpreter within C++ cortex-server code https://docs.python.org/3/extending/embedding.html
2. Run Python app in a separate process -> we probably go with this since it seems easier. For this, there are 2 sub-options
   a. Python<->cortex-webserver with some high-perf IPC (inter-process communication). cortex-server will still serve the HTTP requests
   b. Python has its own webserver (user has to define or we provide?). Again, 2 sub-sub-options
   • External requests will go directly to Python webserver instead of cortex-server (each process must use different ports?)
   • External requests will go to cortex-server first, then cortex-server just route the request to Python webserver

For 2a, I foresee a potential problem: how we will configure/propagate the function signature / API contract from Python to cortex-server.
E.g. Python text-to-speech will expose def tts(text: str) -> bytes (output is encoded audio e.g. mp3) -> how do we avoid hard-coding this in cortex-server code

Case study - Current Python engine

• Dependency management: in a VM, install all required libraries, zip it and ship to HF
  • CI job: https://github.com/janhq/cortex.cpp/blob/76b198347af42e6103c7b725b6df3871e3cbb4e5/.github/workflows/python-venv-package.yml
  • Example of outputs: https://huggingface.co/cortexso/fish-speech/tree/fp16-linux-amd64 (see various branches for various platforms)
• Expose Python engine: each Python app has its own user-defined webserver. cortex-server simply routes request to Python webserver
  • https://github.com/janhq/cortex.cpp/blob/76b198347af42e6103c7b725b6df3871e3cbb4e5/engine/extensions/python-engine/python_engine.cc

Extension point

Using uv, apart from serving a model, we can also use it to run Python CLI programs e.g. https://github.com/janhq/robobench

[ramonpzg]

Managing dependencies and running scripts with uv is a great idea. We'll need to figure out if this is possible to have this happen from within C++ (they explicitly don't expose classes in Python to import uv as a normal library but we could double-check this).

In the "How to Expose the Python Engine" section, we could create a tight integration between the two languages by exposing Python objects and tools as C++ ones and vice-versa using packages like pybind11 and scikit-build-core. This could potentially address the 2a concern on how to propagate signature functions like tts() from Python to C++ and vice-versa.

As @gau-nernst mentioned, running Python processes as separate HTTP servers and having thecortex-server act as a reverse proxy/router would be THE solution worth exploring first (IMO).

Some thoughts on a potential action plan to test this:

1. The cortex-server spawns a Python process via uv run, specifying a temporary port or ports that are dynamically assigned and the cortex-server tracks them in a registry (cortex itself lacks this right now and they have to be manually changed in the .cortexrc or on starting a server with the -p flag)
2. The Python process sends its API schema to cortex-server's registration endpoint, for example /metadata. The schemas will define input/output types and provide validation and routing without having to hardcode stuff (ask users to provide a pydantic object(s) or json object, for example)
3. On termination, the cortex-server removes the model from the routing table.
4. Cortex-server validates incoming requests against the schema before proxying.
   • Modify Drogon routes to resolve URLs to backend Python ports using the registry.
   • Optionally extend to multiple instances of a model.
5. For high throughput use gRPC (therefore the python process has different endpoints for this)

Note: I will update this as I give it more thought

[dan-menlo]

Is it possible to have sandboxed environments? Marimo seems to have done it with uv:

https://github.com/marimo-team/marimo/releases/tag/0.8.4

[gau-nernst]

Run marimo notebooks in package sandboxes. Use
marimo edit --sandbox notebook.py
to edit a Python notebook in a completely isolated virtual environment!

It's just having a separate virtual env to avoid package conflicts (the current Python PR is ready doing this). It's not sandbox in the security sense (Python script shouldn't be allowed to access host files, internet...)

[gau-nernst]

As discussed previously, to limit the scope of this feature, we won't expose Python engine directly but only specific applications. For the current milestone, we aim to add vLLM as an alternative backend for Cortex.