A common pattern is exposing the various bedrock models (Nova, Claude, Lllama3 etc) via API Gateway and Lambda.

[sterankin]

A common pattern is exposing the various bedrock models (Nova, Claude, Lllama3 etc) via API Gateway and Lambda.

Client -> AWS Api Gateway -> AWS Lambda -> Bedrock Model

I noticed this PR: #7828
It mentioned http(s) for Bedrock. Is there a way to call the Bedrock models this way using LiteLLM?

e.g.

from litellm import completion

response = completion(
            model="claude-3-5-sonnet-20241022",
            messages = [{ "content": "Tell me a joke","role": "user"}],
            api_base="https://some-api-url/models",
            api_key="Bearer Token",
)

The above will fail with a 404 as LiteLLM seems to add to the url:

https://some-aws-apigateway-url/bedrock/**v1/messages** when I really want it to be like:

https://some-api-url/models/claude-3-5-sonnet-20241022

The API URL is specific to our infrastructure.

I also tried this:

import os
import litellm
from litellm import completion

litellm.set_verbose = True # 👈 SEE RAW REQUEST

response = completion(
            model="bedrock/converse/us.amazon.nova-pro-v1:0",
            messages=[{ "content": "Hello, how are you?","role": "user"}],
            aws_access_key_id="",
            aws_secret_access_key="",
            aws_region_name="",
            aws_bedrock_runtime_endpoint="https://some-api-url/models",
            extra_headers={"user_id": "1234567", "authorization": "BEARER TOKEN GOES HERE"}

litellm.exceptions.AuthenticationError: litellm.AuthenticationError: BedrockException Invalid Authentication - Unable to locate credentials

But it would be great if we could could use the LiteLLm library to simply make a normal REST call to any api endpoint, and tell it what model or format to expect (e.g. could be Bedrock Converse, or OpenAI format). It should not care about AWS creds it should simply use a Bearer Token.

Originally posted by @sterankin in #7833

@krrishdholakia you had asked me to open this.

[krrishdholakia]

Thank you for the issue @sterankin

if we could could use the LiteLLm library to simply make a normal REST call to any api endpoint, and tell it what model or format to expect

that's a fair point. I think we could just expose a bedrock/converse_like/<model> route

similar to openai_like/

this would leverage the bedrock transformation (bedrock/chat/converse_transformation.py), but skip the aws credential logic (bedrock/chat/converse_handler.py)

[krrishdholakia]

This is now on main @sterankin

from litellm import completion

response = completion(
    model="bedrock/converse_like/some-model",
    messages=[{"role": "user", "content": "What's AWS?"}],
    api_key="sk-1234",
    api_base="https://some-api-url/models",
    extra_headers={"test": "hello world"},
)

will be live in the next release