[FEATURE] Retries for ServiceUnavailableException (503 errors)

[austinmw]

Problem Statement

If I'm reading correctly, it seems that retry handling only captures ThrottlingException; if so, would it be reasonable to capture and retry other intermittent errors like ServiceUnavailableException as well?

Proposed Solution

No response

Use Case

ServiceUnavailableException can happen often with certain models, like Opus 4

Alternatives Solutions

No response

Additional Context

No response

[zastrowm]

Not directly related, but I'm curious if this could be achieved via hooks. We could convert our current retry behavior to something that fires on the AfterModelInvocationEvent and have a retry/retry_delay properties on the event indicating if it should be retried. That would allow callers to configure which exceptions they retry on

[NateLangloisProscia]

I believe this is related, but let me know if not and I'll be happy to open a new issue.

I am using strands (1.1.0) BedrockModel for the strands Agent's model parameter, and I tried using boto_client_config to provide retries to handle 503. It didn't seem to work to handle the retries, since I see exactly 1 invocationError in CloudWatch Bedrock dashboard, and my logs suggest that my code for catching the service unavailable exception still got hit; if it were handling retries, I would have expected to see 3 InvocationServerErrors before hitting my catch block. Here's a snippet[1] and stack trace[2]. Is this expected? If so, why?

[1]

sdk_retry_config = BotoConfig(retries={"total_max_attempts": 3, "mode": "standard"})
model = BedrockModel(
    model_id="your model id",
    boto_client_config=sdk_retry_config,
    **settings.inference_config,
)
agent = Agent(
    model=model,
    system_prompt=system_prompt,
    session_manager=session_manager,
    tools=tools,
    trace_attributes={"correlation.trace_id": trace_id},
)
try:
    result = await agent_wrapper.invoke_async(message)
except Exception as e:
    logger.exception(f"Error during chat: {str(e)}")

[2]

ERROR - Error during chat: An error occurred (serviceUnavailableException) when calling the ConverseStream operation: Bedrock is unable to process your request.
Traceback (most recent call last):
  File "/app/my_app/app.py", line 227, in invoke_async
    return await self.agent.invoke_async(message)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/strands/agent/agent.py", line 398, in invoke_async
    async for event in events:
  File "/app/.venv/lib/python3.12/site-packages/strands/agent/agent.py", line 508, in stream_async
    async for event in events:
  File "/app/.venv/lib/python3.12/site-packages/strands/agent/agent.py", line 544, in _run_loop
    async for event in events:
  File "/app/.venv/lib/python3.12/site-packages/strands/agent/agent.py", line 583, in _execute_event_loop_cycle
    async for event in events:
  File "/app/.venv/lib/python3.12/site-packages/strands/event_loop/event_loop.py", line 187, in event_loop_cycle
    raise e
  File "/app/.venv/lib/python3.12/site-packages/strands/event_loop/event_loop.py", line 132, in event_loop_cycle
    async for event in stream_messages(agent.model, agent.system_prompt, agent.messages, tool_specs):
  File "/app/.venv/lib/python3.12/site-packages/strands/event_loop/streaming.py", line 318, in stream_messages
    async for event in process_stream(chunks):
  File "/app/.venv/lib/python3.12/site-packages/strands/event_loop/streaming.py", line 273, in process_stream
    async for chunk in chunks:
  File "/app/.venv/lib/python3.12/site-packages/strands/models/bedrock.py", line 350, in stream
    await task
  File "/usr/local/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/opentelemetry/instrumentation/threading/__init__.py", line 171, in wrapped_func
    return original_func(*func_args, **func_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/strands/models/bedrock.py", line 447, in _stream
    raise e
  File "/app/.venv/lib/python3.12/site-packages/strands/models/bedrock.py", line 385, in _stream
    for chunk in response["stream"]:
                 ~~~~~~~~^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/botocore/eventstream.py", line 592, in __iter__
    parsed_event = self._parse_event(event)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/botocore/eventstream.py", line 608, in _parse_event
    raise EventStreamError(parsed_response, self._operation_name)
botocore.exceptions.EventStreamError: An error occurred (serviceUnavailableException) when calling the ConverseStream operation: Bedrock is unable to process your request.
└ Bedrock region: us-east-1
└ Model id: anthropic.claude-3-haiku-20240307-v1:0

[NateLangloisProscia]

FWIW, I worked around this with a hopefully short-term hack until this issue is resolved. I extended BedrockModel and had it raise ModelThrottledException when encountering the serviceUnavailableException. This seemed to successfully trigger the retry logic that strands has for throttling exception. Here's a snippet modifying this original code:

-             if e.response["Error"]["Code"] == "ThrottlingException":
+             if (
+                 e.response["Error"]["Code"] == "ThrottlingException"
+                 or "serviceUnavailableException" in error_message
+             ):
                 raise ModelThrottledException(error_message) from e