webui: simple MCP (Model Context Protocol) support (stdio, tool calls)

[ochafik]

Summary

This prototype PR adds MCP (Model Context Protocol) support to llama-server's web ui:

• Tool calls for starters
• Stdio transport only: the server manages subprocesses for the frontend, which connects to them through a WebSocket per conversation per server.
• Tool calls & their results are displayed in same block w/ collapsed inputs & outputs

Note: not meant to merged as is, this will need deduping w/ any existing work (cc/ @allozaur ) / sending in reviewable chunks.

Features

• WebSocket server on HTTP port + 1 for real-time MCP communication
• MCP bridge that spawns and manages MCP server subprocesses (stdio transport)
• Frontend UI for MCP server management with tool exploration
• Tool calling integration in chat completions with streaming support
• Auto-reconnection with exponential backoff for resilience

New CLI Option

# Use default config location (~/.llama.cpp/mcp.json)
./llama-server -m model.gguf

# Or specify config path
./llama-server -m model.gguf --mcp-config /path/to/mcp.json

Configuration Example

{
  "mcpServers": {
    "brave-search": {
      "command": "npx",
      "args": ["-y", "@brave/brave-search-mcp-server", "--transport", "stdio"],
      "env": {
        "BRAVE_API_KEY": "... get your key at https://api.search.brave.com/app/keys ..."
      }
    },
    "python": {
      "command": "uvx",
      "args": ["mcp-run-python", "--deps", "numpy,pandas,pydantic,requests,httpx,sympy,aiohttp", "stdio"],
      "env": {}
    }
  }
}

Architecture

• server-ws.cpp/h - WebSocket server implementation
• server-mcp-bridge.cpp/h - Routes WebSocket connections to MCP subprocesses
• server-mcp.h - MCP protocol type definitions
• Uses sheredom/subprocess.h for cross-platform subprocess management
• Frontend: MCP service, stores, and UI components

API Endpoints

• GET /mcp/servers - List available MCP servers
• WS /mcp?server=<name> - WebSocket connection (on HTTP port + 1)

Test plan

• Unit tests added (tools/server/tests/unit/test_mcp.py)
• Manual testing with @modelcontextprotocol/server-filesystem
• Test tool calling in chat UI
• Test connect/disconnect in MCP picker
• Verify WebSocket reconnection after server restart

TODOs before undrafting

• De-AI slopify this
• Try and use same port for WS (not possible: httplib doesn't support WS)
• Support more tool result types (image, audio, resources, resource links)

Possible follow ups:

• More MCP features: resources, logging, remote servers...
• MCP Apps

[ngxson]

I think @allozaur already working on the same feature (not sure if there is a tracking issue somewhere), probably better to avoid duplicated works

[ngxson]

Just to note that we generally don't review / merge this much backend code at once because it's too risky in term of security. There was recently many instances of data race in the code that even AI can't pick up.

It's recommended to ship backend feature one-by-one, in separated PRs. Or even better, have first version to work without stdio, and add it at later stage.

[ngxson]

I think this is what subprocess.h already doing

[ochafik]

Switched to it, thanks

[ngxson]

Maybe SEE or streamable HTTP will be a more suitable choice if we actually (and seriously) plan to implement this into server, it can be supported with the existing httplib

A custom WS server implementation sounds good on surface, but eventually I don't think we can trust AI to build a HTTP server in C++

[ochafik]

I'll explore shttp as first step, makes sense.

Note that streamable HTTP / SSE come w/ challenges:

• if called directly from the FE, they need proper cors headers, which many I've tried don't seem to have (e.g. Access-Control-Allow-Headers: content-type, mcp-protocol-version and Access-Control-Expose-Headers: Mcp-Session-Id)
• if proxied by server, we might run into issues w/ auth, tbc (e.g. confused deputy problem)

I don't think we can trust AI to build a HTTP server in C++

Yeah only started deslopifying this PR. Even if we don't end up merging this it's an interesting journey :-)

[ngxson]

if called directly from the FE, they need proper cors headers, which many I've tried don't seem to have

Currently, the WS component has only one job is to proxy stdin/out/err through WS. If it's all done in localhost, why do we need CORS? (I assume because you're running as 2 separated ports?)

The SSE / streamable HTTP impl will be very simple as it allow exposing stdio proxy as a simple API endpoint instead of a new port. This will be important for users who already using llama.cpp via a reverse proxy and may not be able to open a new public port.

if proxied by server, we might run into issues w/ auth, tbc (e.g. confused deputy problem)

What kind of auth we are talking about? I may missed something, but llama-server only support API key auth via Authorization header atm. I don't think we support multi-tenant config that can lead to the "confused deputy problem" you mentioned.

[ngxson]

Also, just to clarify my intention about SSE / streamable HTTP: what I mean is that we only use them as the transport layer, meaning its only purpose is to deliver bytes from one place to another (in this case between stdio <--> frontend)

In this point of view, we don't need to be 100% compliant with the MCP specs.

[ochafik]

why do we need CORS

@ngxson That's for the remote MCP endpoints to allow / behave well w/ requests received from web clients, which they are not necessarily tested with (the TypeScript MCP SDK examples don't show how to set those headers for instance)

What kind of auth we are talking about?

MCP auth, to access remote servers that are authenticated w/ oauth. Transparent proxying won't work well (if only because of oauth resource validation)

[ngxson]

I think you still misunderstood my intention. I don't mean we will directly the SSE specs defined by MCP that requires auth and/or specific headers. We don't either need auth because it's a local server, not a remote server as you mentioned.

We will use it only to replace what WS currently does. For your use case, we need WS for bidirectional communication between browser and backend. We can just replace it with a simple SSE implementation for server --> front direction, and simple POST request for the front --> back direction. This is essentially what facebook messenger has been doing before WS was a standard.

I'll make a small PoC on how that works a bit later, but won't be too complicated as we already had all the necessary components the existing c++ code base.

[ngxson]

I implement my idea in ochafik#5

TL;DR:

• Completely removed websocket
• Replaced with SSE + HTTP POST request as mentioned earlier
• Transport served under /mcp endpoint (GET + POST)
• Use a custom transport class on webui to bypass the auth logic - we are running on the same host with the webui, not an external server, there is no need for CORS or auth
• The code can be reduced even further

Benefits are:

• Users already serving llama.cpp via reverse proxy have don't have to change anything - it just works
• SSL is automatically handled by httplib
• C++ code is much simpler than websocket

---

Note: In case we need to use an external host with SSE / streamable HTTP in the future, just need to extend from the revered proxy component in server-models.cpp

[ochafik]

@allozaur feel free to reuse any parts of this PR. I've updated / fixed the most obvious issues, switched to the TS MCP SDK (+ added tests that use the Python MCP SDK), using existing helpers where possible (sha1, base64, subprocess.h), etc.

And Merry Christmas 🎄!

[allozaur]

@allozaur feel free to reuse any parts of this PR. I've updated / fixed the most obvious issues, switched to the TS MCP SDK (+ added tests that use the Python MCP SDK), using existing helpers where possible (sha1, base64, subprocess.h), etc.

And Merry Christmas 🎄!

Thanks, merry Xmas!

Also one note from me just for now is that I don't think we should introduce MCP implementation on the server-side in the first release.

Client-side implementation purely in WebUI is the least invasive and KISS approach that we can have from the beginning. After that we can of course iterate.

[ochafik]

Also one note from me just for now is that I don't think we should introduce MCP implementation on the server-side in the first release.

@allozaur Agree, although the only server-side bit in this PR is the websocket -> stdio passthrough transport, which is at the (core or) edge of the MCP standard w/ no knowledge of what it's transporting (just lines of text both ways). All the protocol level handling (knowledge of MCP messages) is still only on FE side.

If and when we want server-side MCP support, I think we should implement OpenAI's Messages API, or maybe their Realtime API (allowing bits of agent-loops to happen on the server, which would have larger consequences in terms of threading model, etc)