[Feature]: LiteLLM MCP Roadmap

[ishaan-jaff]

The Feature

Starting this to discuss LiteLLM MCP Roadmap

• Allow setting mcp servers on LiteLLM Keys, Teams
• Allow adding, editing, deleting mcp servers on litellm ui
• Allow segmenting tools on /mcp/list using /mcp/list/{server_name}
• Expose a way for users to make an LLM API request and litellm calls /list tools, call tools and sends that to the LLM API
• Add support for certificate auth to MCP SSE servers

Are you a ML Ops Team?

No

Twitter / LinkedIn details

No response

[ishaan-jaff]

cc @msabramo @marty-sullivan @jeromeroussin

[krrishdholakia]

Can this be made into a discussion? @ishaan-jaff

it'll be easier to track

[ishaan-jaff]

done

[olafgeibig]

I think a very common problem is that very quickly the number of MCP servers you have installed explodes. I think that makes prompt engineering much more difficult because you have to carefully tune your prompt that it's not choosing the wrong tools for the task you plan to do. So I think it would be a great feature if the proxy would have several endpoints to offer different subsets of MCP servers that are installed so that you can have one endpoint for your IDE focused on software development tools and another for a general chat client. Just a mechanism to have different profiles with different sets of MCP servers available.

[krrishdholakia]

Could we leverage something like tags here? https://docs.litellm.ai/docs/proxy/tag_routing

[msabramo]

I wonder if you should leave room for the possibility of doing this, but not invest in it right away. LLMs are pretty smart and I don't know that picking the wrong tools would happen that often unless you have a lot of very similar tools (maybe some people have that use case; I don't see that being an issue for me). Maybe try it and then solve if that pops up a lot? But make sure whatever you do API-wise leaves room for that possibility. Maybe you can throw all_tools or something into the path of the request so you have a place where you can specify a tool collection if needed?

[msabramo]

Another thought off the top of my head - what if there was a way to specify which tools are automatically used based on prompt language and which aren't (and have to be explicitly requested)?

That lets you specify that say the "check deployment status" tool can be invoked automatically but the "do deployment tool" has to be explicitly requested so there are no accidental deploys.

Could allow that on the tool level or if you end up doing tool collections, it could be specified on the tool collection level - i.e.: idempotent_tools and non_idempotent_tools.

[olafgeibig]

@ishaan-jaff Yes, now I get it. That would probably do the trick. At least on a client level. I think many users would like to remove the necessity to manage multiple sets MCPs for different clients. But have one central place to manage and maintain the MCP installations while still be able to have different sets of MCPs for different applications.

[marty-sullivan]

I like the "segmentation" idea too. If you can narrow down the types of tools you want to expose to the LLM, that can only help.

Allow setting mcp servers on LiteLLM Keys, Teams

This is also a good one. It would definitely be good to be able to limit access to certain MCPs by org/team membership.