(local) Ollama as AI backend

[WuerfelDev]

Is your feature request related to a problem? Please describe.

• i don't want to create an online account if its not necessary
• I might have sensitive data
• Company might not allow online AI models

Describe the solution you'd like
An option to connect to a local Ollama instead of using OpenRouter

Describe alternatives you've considered
-

Additional context
https://github.com/ollama/ollama/blob/main/docs/api.md

[alvinunreal]

Updated readme to include instructions for local Ollama.
This would be example config:

openrouter:
  api_key: api-key
  model: gemma3:1b
  base_url: http://localhost:11434/v1

but it's not tested with it - might need some prompt tuning.
If anyone is able to test and contribute learnings.

[ductng]

Hi,

I just tried with my local installation, but I get 404 errors:
Failed to get response from AI: API returned error: 404 page not found

I am not sure, if the Ollama API is the same as the OpenAI API.

[drzhouq]

same issue. I can confirm that the end point is actually working well:
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json"
-d '{
"model": "gemma3:1b",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Hello!"
}
]
}'
{"id":"chatcmpl-826","object":"chat.completion","created":1746030753,"model":"gemma3:1b","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"Hello there! How can I help you today? 😊"},"finish_reason":"stop"}],"usage":{"prompt_tokens":22,"completion_tokens":12,"total_tokens":34}}

[WuerfelDev]

It worked after removing the tailing slash from the url in the config file (golang/go#69063). However when using ollama's OpenAi-compatible API there is no way to set the prompt input limit, which is set to 2048 token and thats way too little.
There even is a pr at ollama but its been there since 10 months already...

The best way would probably be to use the ollama go package (https://pkg.go.dev/github.com/ollama/ollama/api)
I haven't looked into how feasible it is to implement it as a second backend in tmuxai

[alvinunreal]

Nice find!

[Lippiece]

I may be mistaken, but isn't it because tmuxai uses GET?

[GIN] 2025/05/04 - 13:13:14 | 200 |  3.066065487s |       127.0.0.1 | POST     "/v1/chat/completions"
[GIN] 2025/05/04 - 13:13:32 | 404 |       6.197µs |       127.0.0.1 | GET      "/v1/chat/completions"

[WuerfelDev]

I may be mistaken, but isn't it because tmuxai uses GET?

It does not use GET. But for some reason go's http library transforms the request to GET when there is a double slash in the url. So make sure your url in the config ends without a slash
https://github.com/alvinunreal/tmuxai/blob/main/internal%2Fai_client.go#L103

[alvinunreal]

Double slash issue should be fixed, just released new version

[alvinunreal]

ollama/ollama#10366

There is also this issue with gemma, can you try other models?

[AbnerTeng]

Hi,

I just tried with my local installation, but I get 404 errors: Failed to get response from AI: API returned error: 404 page not found

I am not sure, if the Ollama API is the same as the OpenAI API.

Do you use localhost:11434 or localhost:11434/v1? The second one must work

[sub2010]

I dont get it! Ollama has no api key. And even i try some random api keys it doesnt work. Do you have some tip?

ollama:
model: gemma3:1b
base_url: http://localhost:11434

[AbnerTeng]

I mean that you need to change your base_url from http://localhost:11434 to http://localhost:11434/v1 for RESTful API structure, since OpenAI follows it.

[sub2010]

Thank you, i tested so many things. That i stop using my brain. Now it works :)