Enable more advanced Anthropic prompt caching than `CachePoint` implementation llows

[nathan-gage]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

I think it would be worth reverting the CachePoint implementation of Anthropic Context Caching.

It has caused several problems regarding message serialization, specifically. In order to use context caching for traditional chat conversations, we seemingly need a message history processor to remove CachePoint from previously serialized messages. Otherwise, I've enabled anthropic_cache_instructions and anthropic_cache_tool_definitions, with a CachePoint after the user_prompt, and am seeing ModelHTTPErrors: 'A maximum of 4 blocks with cache_control may be provided. Found 7.'

While I understand *why* I'm getting those, the DX is not great. Here's an MRE.

from pydantic_ai import Agent, RunContext, CachePoint, AgentRunResultEvent
from pydantic_ai.models.anthropic import AnthropicModelSettings
from dataclasses import dataclass

@dataclass
class MREDeps:
    flag: bool

mre = Agent(
    "anthropic:claude-sonnet-4-5-20250929",
    model_settings=AnthropicModelSettings(
        max_tokens=64_000,
        anthropic_cache_instructions=True,
        anthropic_cache_tool_definitions=True,
    )
)


@mre.instructions
def instructions(ctx: RunContext[MREDeps]) -> str:
    if ctx.deps.flag:
        return "Talk in piglatin"
    return "Talk in a Texan accent"


@mre.tool
def yeehaw(ctx: RunContext[MREDeps], phrase: str) -> str:
    if not ctx.deps.flag:
        return f"yee was haw'd: {phrase.upper()}"
    return "ERROR: you're not talkin' texan"


messages = []

for i in range(10):
    user_text = input("> ")
    print("User input:", user_text)
    
    async for event in mre.run_stream_events([user_text, CachePoint()], message_history=messages, deps=MREDeps(i % 2 != 0)):
        if isinstance(event, AgentRunResultEvent):
            new_messages = event.result.new_messages()
            messages.extend(new_messages)

            for m in new_messages:
                print(m)

And the logs:

User input: howdy partner
ModelRequest(parts=[UserPromptPart(content=['howdy partner', CachePoint(kind='cache-point')], timestamp=datetime.datetime(2025, 11, 17, 17, 28, 1, 509238, tzinfo=datetime.timezone.utc))], instructions='Talk in a Texan accent', run_id='1a6bba3c-b7c7-439b-8974-b4bfa0cbb68b')
ModelResponse(parts=[TextPart(content="Howdy there, partner! Well ain't this a fine day! Welcome on in, friend. What can I do ya for today? Y'all need any help with somethin', or just stoppin' by to shoot the breeze? Either way, I'm happier than a tornado in a trailer park to see ya!")], usage=RequestUsage(input_tokens=564, output_tokens=72, details={'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 0, 'input_tokens': 564, 'output_tokens': 72}), model_name='claude-sonnet-4-5-20250929', timestamp=datetime.datetime(2025, 11, 17, 17, 28, 4, 483811, tzinfo=datetime.timezone.utc), provider_name='anthropic', provider_details={'finish_reason': 'end_turn'}, provider_response_id='msg_01BTNejiLTFEAjySC2vg4Uad', finish_reason='stop', run_id='1a6bba3c-b7c7-439b-8974-b4bfa0cbb68b')
User input: can you try yee'in
ModelRequest(parts=[UserPromptPart(content=["can you try yee'in", CachePoint(kind='cache-point')], timestamp=datetime.datetime(2025, 11, 17, 17, 28, 12, 195243, tzinfo=datetime.timezone.utc))], instructions='Talk in piglatin', run_id='2a2e4a1f-13ec-4f22-8093-d53a689f01b0')
ModelResponse(parts=[TextPart(content='Owday-hay, artnerpay! Ime-tay otay ive-gay is-thay a-yay ot-shay!'), ToolCallPart(tool_name='yeehaw', args='{"phrase": "Howdy partner! Time to give this a shot!"}', tool_call_id='toolu_018tKuUA2PnPex69W1kvANfz')], usage=RequestUsage(input_tokens=644, output_tokens=100, details={'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 0, 'input_tokens': 644, 'output_tokens': 100}), model_name='claude-sonnet-4-5-20250929', timestamp=datetime.datetime(2025, 11, 17, 17, 28, 16, 669363, tzinfo=datetime.timezone.utc), provider_name='anthropic', provider_details={'finish_reason': 'tool_use'}, provider_response_id='msg_01LNaaEPX9vWCkR2tn2iyYwr', finish_reason='tool_call', run_id='2a2e4a1f-13ec-4f22-8093-d53a689f01b0')
ModelRequest(parts=[ToolReturnPart(tool_name='yeehaw', content="ERROR: you're not talkin' texan", tool_call_id='toolu_018tKuUA2PnPex69W1kvANfz', timestamp=datetime.datetime(2025, 11, 17, 17, 28, 19, 4602, tzinfo=datetime.timezone.utc))], instructions='Talk in piglatin', run_id='2a2e4a1f-13ec-4f22-8093-d53a689f01b0')
ModelResponse(parts=[TextPart(content='Uh-oh-yay! Ooks-lay ike-lay I-yay ot-gay old-tay off-yay or-fay ot-nay alkin-tay exan-Tay enough-yay! Et-lay e-may y-tray at-thay again-yay ith-way ore-may outhern-Say air-flay!'), ToolCallPart(tool_name='yeehaw', args='{"phrase": "Well howdy there partner! Yeehaw, ain\'t this just a rootin\' tootin\' good time! Y\'all come back now, ya hear?"}', tool_call_id='toolu_014ymJHfnVd11MjGr1CrzDX1')], usage=RequestUsage(input_tokens=766, output_tokens=183, details={'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 0, 'input_tokens': 766, 'output_tokens': 183}), model_name='claude-sonnet-4-5-20250929', timestamp=datetime.datetime(2025, 11, 17, 17, 28, 21, 256867, tzinfo=datetime.timezone.utc), provider_name='anthropic', provider_details={'finish_reason': 'tool_use'}, provider_response_id='msg_01L9i4aXTkPHe7YRtfcB6UF6', finish_reason='tool_call', run_id='2a2e4a1f-13ec-4f22-8093-d53a689f01b0')
ModelRequest(parts=[ToolReturnPart(tool_name='yeehaw', content="ERROR: you're not talkin' texan", tool_call_id='toolu_014ymJHfnVd11MjGr1CrzDX1', timestamp=datetime.datetime(2025, 11, 17, 17, 28, 26, 136516, tzinfo=datetime.timezone.utc))], instructions='Talk in piglatin', run_id='2a2e4a1f-13ec-4f22-8093-d53a689f01b0')
ModelResponse(parts=[TextPart(content='Oot-shay! is-Thay is-yay ougher-tay an-thay a-yay o-tway-ollar-day eak-stay! Et-lay e-may y-tray one-yay ore-may ime-tay ith-way even-yay ore-may exan-Tay irit-spay!'), ToolCallPart(tool_name='yeehaw', args='{"phrase": "YEEHAW! Howdy partner! Well I\'ll be hornswoggled! This here\'s finer than frog hair split four ways! Hot diggity dog, ain\'t nothin\' better than ridin\' off into the sunset with good folks!"}', tool_call_id='toolu_01KH9LZoUp5TzMY36z8g4jQD')], usage=RequestUsage(input_tokens=971, output_tokens=196, details={'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 0, 'input_tokens': 971, 'output_tokens': 196}), model_name='claude-sonnet-4-5-20250929', timestamp=datetime.datetime(2025, 11, 17, 17, 28, 29, 910374, tzinfo=datetime.timezone.utc), provider_name='anthropic', provider_details={'finish_reason': 'tool_use'}, provider_response_id='msg_01THAPikxEV21Uyqfzw5ypRB', finish_reason='tool_call', run_id='2a2e4a1f-13ec-4f22-8093-d53a689f01b0')
ModelRequest(parts=[ToolReturnPart(tool_name='yeehaw', content="ERROR: you're not talkin' texan", tool_call_id='toolu_01KH9LZoUp5TzMY36z8g4jQD', timestamp=datetime.datetime(2025, 11, 17, 17, 28, 33, 539040, tzinfo=datetime.timezone.utc))], instructions='Talk in piglatin', run_id='2a2e4a1f-13ec-4f22-8093-d53a689f01b0')
ModelResponse(parts=[TextPart(content="Ell-way oot-shay, artnerpay! Eems-say ike-lay I-yay ust-jay an-cay't-yay et-gay e-thay ight-ray exas-Tay ang-tway o-tay is-thay ere-hay unction-fay! Ight-may e-bay it-yay ants-way even-yay ore-may authentic-yay exan-Tay alk-tay an-thay I-yay an-cay uster-may up-yay!")], usage=RequestUsage(input_tokens=1189, output_tokens=141, details={'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 0, 'input_tokens': 1189, 'output_tokens': 141}), model_name='claude-sonnet-4-5-20250929', timestamp=datetime.datetime(2025, 11, 17, 17, 28, 35, 693712, tzinfo=datetime.timezone.utc), provider_name='anthropic', provider_details={'finish_reason': 'end_turn'}, provider_response_id='msg_01PMuFuiujKjc4satcmhw3z5', finish_reason='stop', run_id='2a2e4a1f-13ec-4f22-8093-d53a689f01b0')
User input: darn
Traceback (most recent call last):
  File "/Users/ngage/repos/sweetspot/sweetspot-gov/.venv/lib/python3.11/site-packages/pydantic_ai/models/anthropic.py", line 336, in _messages_create
    return await self.client.beta.messages.create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ngage/repos/sweetspot/sweetspot-gov/.venv/lib/python3.11/site-packages/anthropic/resources/beta/messages/messages.py", line 2430, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/Users/ngage/repos/sweetspot/sweetspot-gov/.venv/lib/python3.11/site-packages/anthropic/_base_client.py", line 1902, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ngage/repos/sweetspot/sweetspot-gov/.venv/lib/python3.11/site-packages/anthropic/_base_client.py", line 1702, in request
    raise self._make_status_error_from_response(err.response) from None
anthropic.BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'A maximum of 4 blocks with cache_control may be provided. Found 5.'}, 'request_id': 'req_011CVDrdJxB6RtskNNdrL48L'}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/ngage/repos/sweetspot/sweetspot-gov/.venv/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 3670, in run_code
    await eval(code_obj, self.user_global_ns, self.user_ns)
  File "/var/folders/0d/y0_v_0415vl8c0dqjchgkxtc0000gn/T/ipykernel_64417/4139601478.py", line 39, in <module>
    async for event in mre.run_stream_events([user_text, CachePoint()], message_history=messages, deps=MREDeps(i % 2 != 0)):
  File "/Users/ngage/repos/sweetspot/sweetspot-gov/.venv/lib/python3.11/site-packages/pydantic_ai/agent/abstract.py", line 906, in _run_stream_events
    result = await task
             ^^^^^^^^^^
  File "/Users/ngage/repos/sweetspot/sweetspot-gov/.venv/lib/python3.11/site-packages/pydantic_ai/agent/abstract.py", line 883, in run_agent
    return await self.run(
           ^^^^^^^^^^^^^^^
  File "/Users/ngage/repos/sweetspot/sweetspot-gov/.venv/lib/python3.11/site-packages/pydantic_ai/agent/abstract.py", line 243, in run
    async with node.stream(agent_run.ctx) as stream:
  File "/Users/ngage/.pyenv/versions/3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ngage/repos/sweetspot/sweetspot-gov/.venv/lib/python3.11/site-packages/pydantic_ai/_agent_graph.py", line 434, in stream
    async with ctx.deps.model.request_stream(
  File "/Users/ngage/.pyenv/versions/3.11.14/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ngage/repos/sweetspot/sweetspot-gov/.venv/lib/python3.11/site-packages/pydantic_ai/models/anthropic.py", line 255, in request_stream
    response = await self._messages_create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ngage/repos/sweetspot/sweetspot-gov/.venv/lib/python3.11/site-packages/pydantic_ai/models/anthropic.py", line 356, in _messages_create
    raise ModelHTTPError(status_code=status_code, model_name=self.model_name, body=e.body) from e
pydantic_ai.exceptions.ModelHTTPError: status_code: 400, model_name: claude-sonnet-4-5-20250929, body: {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'A maximum of 4 blocks with cache_control may be provided. Found 5.'}, 'request_id': 'req_011CVDrdJxB6RtskNNdrL48L'}

Furthermore, since CachePoint is only included in UserPromptPart, its not possible to Cache at user & assistant messages in the chain. While many people dislike Anthropic's impl., the ability to cache at Assistant message parts is a very valuable -- in Anthropic, it is valid to start a conversation with an Assistant message pre-fill.

I'm very much not a fan of this implementation also requiring system/tool_def to be cached at the ModelSettings level. Let's say you have:

tools=[
    StaticTool1(),
    StaticTool2(),
    DynamicTool(),
]

The current impl. does not allow a cache point at StaticTool2, which makes the anthropic_cache_tool_definitions not very useful. This also applies for MCP toolsets.

---

Last I understood, the Context Caching impl. was to be apart of the vendor_details. What happened to that? I vastly prefered the vendor_details or even history_processors based approach.

Python, Pydantic AI & LLM client version

Python 3.11.14
Pydantic 1.18.0
Anthropic 0.72.0

[nathan-gage]

I guess, to distill my concerns, this API is in a weird middle ground of 'developer controlled' and 'magically applied'. If pydantic-ai is going to serialize the CachePoint, then let me control it entirely, but if ModelSettings has a magical anthropic_cache_*, then I would expect the CachePoint in message_history to automatically drop extraneous CachePoint.

[DouweM]

@nathan-gage Thanks for filing the issue; I'm also having some second thoughts about the implementation.

It's in a weird middle ground because only Anthropic supports the feature, there's no indication that OpenAI or Google will add it, and I could imagine Anthropic eventually improving their implicit prompt caching to the point where they stop recommending people to use explicit cache control anymore. So:

• we can't come up with a nice generic implementation of the concept that works across providers
• it doesn't seem worth it to pepper cache related things across the framework to support it for just one provider
• we also can't not support it because Anthropic is one of the big three

Let me go through your points 1 by 1 though and see if there's a reasonable solution other than reverting the feature entirely (and causing issues for people who already upgraded since Friday):

ModelHTTPErrors: 'A maximum of 4 blocks with cache_control may be provided. Found 7.'

Ay, I had been under the mistaken impression that Anthropic would ignore any cache_controls except for the last 4 🤦🏻

At the very least I think we should do that filtering AnthropicModel message post-processing; that shouldn't be too hard to do.

Furthermore, since CachePoint is only included in UserPromptPart, its not possible to Cache at user & assistant messages in the chain. While many people dislike Anthropic's impl., the ability to cache at Assistant message parts is a very valuable -- in Anthropic, it is valid to start a conversation with an Assistant message pre-fill.

How are you doing prefill with Pydantic AI currently? We don't really support it currently (see #836, #2778), although I agree it can be useful.

We could support CachePoint in ModelResponse.parts; would that be sufficient?

I'm very much not a fan of this implementation also requiring system/tool_def to be cached at the ModelSettings level. Let's say you have:

tools=[
StaticTool1(),
StaticTool2(),
DynamicTool(),
]
The current impl. does not allow a cache point at StaticTool2, which makes the anthropic_cache_tool_definitions not very useful. This also applies for MCP toolsets.

I agree that's an annoying limitation. Would your suggested solution be to support CachePoint inside the tools and toolsets lists?

That's the kind of "disruptive", framework-wide-impact change I'm hesitant to make.

My goal with the feature as it was merged was not to claim complete support for Anthropic's prompt caching, but rather to find a reasonable balance between supporting enough that it'd be sufficient for most use cases, and not steering the whole framework in a particular direction just to line up with one provider's particular approach to caching (even if that one provider is Anthropic).

But it's very possible we haven't found the right balance yet, or there's a less disruptive way to achieve the necessary flexibility.

Last I understood, the Context Caching impl. was to be apart of the vendor_details.

Remind me where you saw that? We landed on this CachePoint approach in mid August: #1041 (comment)

[DouweM]

Also check out #3441 / #3442 where people are continuing down this model settings path already. Curious about your thoughts on that as well.

[nathan-gage]

@DouweM thanks for the response.

I totally agree regarding the 'nice generics' problem. That's part of the reason I'm not a big fan of CachePoint; its obvious no providers are adopting this standard and I imagine Anthropic is working to implement implicit caching. Yet, it has very valuable uses, which we rely on heavily, so I was a bit disappointed to see such a opinionated primitive introduced to UserPromptPart & the anthropic model settings. We can't exactly use it, but its somehow still intrusive. And I believe it breaks semantics of what a 'Part' really should be.

How are you doing prefill with Pydantic AI currently? We don't really support it currently (see #836, #2778), although I agree it can be useful.

We do not use pydantic-ai for assistant prefill, currently. We heavily use it for certain workflows, directly via AsyncAnthropic beta client, and were hoping to eventually migrate those to Pydantic-AI, but it theoretically would be impossible with the current CachePoint implementation.

I agree that's an annoying limitation. Would your suggested solution be to support CachePoint inside the tools and toolsets lists?

That's the kind of "disruptive", framework-wide-impact change I'm hesitant to make.

Yea, I agree, I do not believe adding CachePoint to tools/toolsets would be a good change. In general, our current internal message serialization/streaming code would then need to be updated to handle something which is not really part of the user<>assistant interaction lifecycle, as well as our internal MCP registry system. In general, a lot of our code against pydantic-ai would need to handle this explicitly in ways that are unintuitive.

Remind me where you saw that? We landed on this CachePoint approach in mid August: #1041 (comment)

I honestly have no idea, I searched around and couldn't find it. For some reason I remember seeing it somewhere...

Also check out #3441 / #3442 where people are continuing down this model settings path already. Curious about your thoughts on that as well.

IMO, #3441/#3442 have the same problems as the the anthropic_cache_instructions/anthropic_cache_tool_definitions -- It leaves far too much assumption to the pydantic-ai side and not enough control in developers' hands. Personally, I strongly believe that message history altering configuration should not belong in the settings. If the alternative--CachePoint in tools/toolsets--is disruptive then that suggests to me that CachePoint as a concept is flawed.

---

I wish I had the time to submit a proper PR on this issue, currently, we subclass AnthropicModel which overloads the _map_messages function and adds a cache block to the end of the model messages upon request.

If I were to implement this for pydantic-ai on a broader level, I would've added a vendor_request_details/vendor_response_details field to the various *Part dataclasses & ToolDefinition, use those for setting context cache blocks, and adapt the AnthropicModel to prune all cache blocks except the last 4 (or, if less magic is preferred, it could be done history_processor style).

This would allow me to write history processors against the ModelMessages / prepare functions for ToolDefinitions, which behave in an reasonable & expected way. Then pydantic-ai could provide a standard anthropic_cache_last history processor function. The only caveat is that the instructions decorator might need to be updated to allow returning a TextPart as well as str.

I prefer the vendor_*_details approach as it would also open the door for the Anthropic citations API, which we also use heavily (outside of pydantic-ai), as well as some other Gemini-specific oddities (like their explicit context cache implementation, which is extremely underrated and very powerful).

---

As it stands, we'll continue to use our AnthropicModel subclass as I do not want to begin serializing CachePoint to storage nor requiring downstream consumers of the ModelMessage / StreamEvents in our codebase to regularly handle a primitive which is irrelevant to the user<>assistant interaction lifecycle.

[DouweM]

@nathan-gage I had a chance to talk about this a bit with @dmontagu, both the current implementation (and option of reverting it) and possible alternatives. Unfortunately, unsatisfying though we realize the current implementation is in its limited scope (and bugs that will need fixing), we don't see an "obviously better" approach that enables more fine-grained control without bleeding significantly more into the framework (see below).

So I'm afraid the subclass approach is going to continue being the recommended one for more advanced use cases like yours that require more flexibility/power.

Essentially we're OK with the native implementation being more limited, considering that it's a one-provider feature that feels like it will be deprecated at some point, especially as the alternative solution of subclassing is a valid way to "hook" into aspects of model behavior like message processing. (We're also open to more flexible ways of hooking into model classes's Pydantic AI messages -> model API messages transformation.)

I agree CachePoint also doesn't feel like the complete solution here (e.g. it would feel very awkward to have in a tools list), but I'll note that I copied it from the Bedrock Converse API (which supports it just for Anthropic), and it does feel like good "value for money" in terms of implementing a lot of the power of this feature at the cost of just "1 new type, 1 union type update, and 2 new model settings".

(And for what little it's worth in such matters of preference, when I sketched the Anthropic and Bedrock Converse approaches to David, before telling him which one Pydantic AI had ended up with, he said he liked the latter much better!)

---

In any case, I'm definitely still open to improvements (including a way to set details/metadata on arbitrary parts that we could leverage for prompt caching), but for the moment we decided that the current implementation is not "wrong" enough to be reverted, even if it has some (fixable) issues and (acceptable, to us) limitations. (Worst case, we end up with 2 implementations and deprecate one in v2)

So let's talk vendor_*_details. Would you expect the attributes to be sent through to the API verbatim on the part in question, or for the keys to be meaningful and possibly handled (differently) by different model classes?

• If it's verbatim, that's problematic because it means parts/history won't be portable across models/providers anymore, which would break things like FallbackModel as well. (You can work around that in a history processor, but that'd be pretty painful for "regular" users that want to tag a part as cacheable).
• It also seems desirable that if the "this is a cache point" field is set, that would be picked up on by AnthropicModel, BedrockConverseModel, and any future APIs that support the feature (with different API shapes).

So I think it would need to be more like ModelSettings (i.e. a TypedDict with fields implemented by multiple model classes, and model-specific subclasses with prefixed fields) so that it can be handled type-safely and portably.

I can see a use for that in other places as well (like image parts that can take detail: high|low on OpenAI, which may also be supported by other models at some point with a different name), but that opens a pandora's box of dedicated TypedDicts for each part's details based on which subset of fields applies to them... (I also haven't seen a use for fields like this on tool defs and instructions yet, outside of caching, but I'd be fine saying "if we do it, let's do it on everything".)

I think we'll have to go into that direction eventually (for example for citations: #3126), but its a lot more complex than the current solution (or your subclass solution for that matter) and hard to justify changes to probably at least a dozen part types.

[Wh1isper]

Last I understood, the Context Caching impl. was to be apart of the vendor_details. What happened to that? I vastly prefered the vendor_details or even history_processors based approach.

It sounds like we're missing a hook for Pydantic Tool List -> Model Tool List, similar to history_processor, as a more powerful alternative to prepare_tools or Toolsets(Maybe it's a kind of Toolsets?).

ModelSettings often serve as a quick configuration way, which doesn't conflict with giving developers more control and I favor approaches like the history_processor for dynamic tools.

[nathan-gage]

@DouweM Thanks for the deliberation. That totally makes sense -- I understand that my use is a lot harder to generalize to the rest of the library!

My biggest concern, looking forward, is the proliferation of ModelSettings flags for various CachePoint placements -- if the behavior changes at all, it is non-obvious and critical implementation detail that's difficult to surface by the caller. Depending on where I'm hoping to place my CachePoints, I would need to read through the source code. Furthermore, if Pydantic drops the earliest CachePoints, something like anthropic_cache_tool_definitions may never be applied, even though I explicitly flagged it as True.

The easy alternative I see is providing a standard history processor or Model subclass, leaving the details behind a single go-to definition (which would also serve as a good reference for developers needing more specific implementations). While similarly inflexible as the CachePoint-style, it's at least a bit more explicit, and doesn't require Pydantic AI to change its CachePoint parsing if Anthropic relaxes the 4 cache control block restriction--maintainers would only need to change the history processor/Model implementation directly. As I see it, the CachePoint style is equally inflexible but has the added downside of the various ModelSettings flags being proposed.

---

Another problem to address is less obvious: the CachePoint implementation assumes that the CachePoint must be applied to the UserPromptParts in the immediate Agent.request_stream call thereafter, which fails for the 'message editing' use-case.

Because the CachePoint is applied to the user request, there's no way to apply a CachePoint at the end of a long Agent run, and so, if the user edits their message, we lose the benefit of a cache for the entire Agent run.

Example:

- User 1st prompt
- CachePoint
- < many agent rounds >
- User's 2nd prompt
- CachePoint

Editing 'User's 2nd prompt' will result in no cached tokens for the '< many agent rounds >'.

We're also open to more flexible ways of hooking into model classes's Pydantic AI messages -> model API messages transformation.

This alone would be a great ergonomic change -- it is a bit annoying updating the subclass to match the function when it changes. We recently ran into an issue where we didn't realize the beta-param handling was changed slightly, and caused a minor regression on Anthropic usage in prod.

---

vendor_*_details

I partially disagree with the idea that verbatim parts would not be portable, as I'd imagine they'd be dropped when mapping to the request surface, if unsupported; it would likely look like openai_detail and anthropic_cache_control -- it's up to the Model implementation to handle those, if at all.

Perhaps this is roughly what you mean by the TypedDict style implementation...

I'd imagine this is a useful construct as it would allow you to ship details alongside *Parts, which is currently not possible.

In this situation, TypedDict is moreso for type-hinting well-known provider details, but everything is processed or forwarded using the provider key prefix. The maintenance would look more like KnownModelName rather than ModelSettings, I guess.

Given that this is more one-off than defining a standard for cache markers, it would be acceptable that AnthropicModel & BedrockConverseModel would both need to implement their own specific cache vendor details. The Converse API caching for nova models actually works slightly differently than the Anthropic-spec Bedrock Converse implementation, so differentiating those is beneficial in certain circumstances (nova provides implicit caching, so the explicit cache markers can break performance of implicit caching if done improperly).

Largely this approach makes sense for citation-like APIs, which we've personally found a lot of value in, so it is a bit forward looking in that regard.

[DouweM]

My biggest concern, looking forward, is the proliferation of ModelSettings flags for various CachePoint placements -- if the behavior changes at all, it is non-obvious and critical implementation detail that's difficult to surface by the caller. Depending on where I'm hoping to place my CachePoints, I would need to read through the source code. Furthermore, if Pydantic drops the earliest CachePoints, something like anthropic_cache_tool_definitions may never be applied, even though I explicitly flagged it as True.

@nathan-gage Yeah I don't love the proliferation of settings, especially as they don't actually map to API attributes, but I don't see that as that much of a problem as they're Anthropic-specific and there's only 2 (+ #3442).

I think that for the typical user who thinks "my tool defs/instructions are static so they should be cached if possible" and doesn't feel like they need to know the ins and out of explicit prompt caching, these flags are a lot more convenient than message processing, even if (like implicit caching) it's less powerful, depends on heuristics, and won't always give the optimal behavior. As mentioned, this feature was mostly meant to cater to them, not those who need that full control.

But I'm curious about the problems you foresee "if the behavior changes at all": why would we need to change those those model settings work? (Even if we do, I'd consider that OK as long as stick to the "spirit of the law" in terms of how they were documented)

The easy alternative I see is providing a standard history processor or Model subclass, leaving the details behind a single go-to definition (which would also serve as a good reference for developers needing more specific implementations). While similarly inflexible as the CachePoint-style, it's at least a bit more explicit, and doesn't require Pydantic AI to change its CachePoint parsing if Anthropic relaxes the 4 cache control block restriction--maintainers would only need to change the history processor/Model implementation directly. As I see it, the CachePoint style is equally inflexible but has the added downside of the various ModelSettings flags being proposed.

That processing would need to happen at the API messages level rather than Pydantic AI messages, so a history processor wouldn't really work. But we can definitely move some methods around on AnthropicModel to make it easier to override only the relevant bits, and then document those hooks with an example subclass.

Another problem to address is less obvious: the CachePoint implementation assumes that the CachePoint must be applied to the UserPromptParts in the immediate Agent.request_stream call thereafter, which fails for the 'message editing' use-case.

Because the CachePoint is applied to the user request, there's no way to apply a CachePoint at the end of a long Agent run, and so, if the user edits their message, we lose the benefit of a cache for the entire Agent run.

Example:

- User 1st prompt
- CachePoint
- < many agent rounds >
- User's 2nd prompt
- CachePoint

Editing 'User's 2nd prompt' will result in no cached tokens for the '< many agent rounds >'.

Would allowing CachePoint on ModelResponse.parts help with that?

We're also open to more flexible ways of hooking into model classes's Pydantic AI messages -> model API messages transformation.

This alone would be a great ergonomic change -- it is a bit annoying updating the subclass to match the function when it changes. We recently ran into an issue where we didn't realize the beta-param handling was changed slightly, and caused a minor regression on Anthropic usage in prod.

I'll gladly take a contribution to AnthropicModel to make it easier to override only the relevant bits!

vendor_*_details

I partially disagree with the idea that verbatim parts would not be portable, as I'd imagine they'd be dropped when mapping to the request surface, if unsupported; it would likely look like openai_detail and anthropic_cache_control -- it's up to the Model implementation to handle those, if at all.

Perhaps this is roughly what you mean by the TypedDict style implementation...

Yeah, pretty much. detail is a good example, as it started as an OpenAI-only feature, but now with Gemini 3 Google added media_resolution with the same behavior, so in #3464 I'm making detail map to that as well.

People's expectation of the framework is that they define what they want just once, and we do the heavy lifting of translating that to each model API, so if something is supported by multiple APIs, I'd much prefer to have just a single key than one or more prefixed ones.

I'd imagine this is a useful construct as it would allow you to ship details alongside *Parts, which is currently not possible.

In this situation, TypedDict is moreso for type-hinting well-known provider details, but everything is processed or forwarded using the provider key prefix. The maintenance would look more like KnownModelName rather than ModelSettings, I guess.

I agree that if we already had a vendor_*_details, this whole discussion would be a lot easier because we'd just have added anthropic_cache_control and then see out about other APIs later or never. So I wouldn't mind going in that direction to make similar things easier in the future.

Given that this is more one-off than defining a standard for cache markers, it would be acceptable that AnthropicModel & BedrockConverseModel would both need to implement their own specific cache vendor details. The Converse API caching for nova models actually works slightly differently than the Anthropic-spec Bedrock Converse implementation, so differentiating those is beneficial in certain circumstances (nova provides implicit caching, so the explicit cache markers can break performance of implicit caching if done improperly).

Ah that's good to know. Do you have any opinions on #3438?

Largely this approach makes sense for citation-like APIs, which we've personally found a lot of value in, so it is a bit forward looking in that regard.

I was planning to implement that using proper dataclasses abstracting over everything the big three support, rather than having provider-specific typed dicts. Would love your thoughts on #3126.

[nathan-gage]

@DouweM

But I'm curious about the problems you foresee "if the behavior changes at all": why would we need to change those those model settings work? (Even if we do, I'd consider that OK as long as stick to the "spirit of the law" in terms of how they were documented)

Mostly related to things like cache_ttl or a cache_control which is no longer type: 'ephemeral'. But its a bit pedantic, on second thought.

Would allowing CachePoint on ModelResponse.parts help with that?

Probably, but I imagine it to be ergonomically awkward, as you couldn't do Agent.run([CachePoint(), "< user prompt >"], ...). Unless Model would try to apply it to the right-most allowed part to the before the CachePoint.

I'll gladly take a contribution to AnthropicModel to make it easier to override only the relevant bits!

We'll see if I have the free time hahahaha. I would love to make this happen though.

People's expectation of the framework is that they define what they want just once, and we do the heavy lifting of translating that to each model API, so if something is supported by multiple APIs, I'd much prefer to have just a single key than one or more prefixed ones.

This is actually very good perspective into your philosophy for Pydantic-AI... it makes all of this make more sense. I must say -- I'm totally understanding of the decisions here, I realize I sounded a bit combative -- was just providing some perspective as a heavy Pydantic AI & Anthropic user, but this is all very very reasonable.

Ah that's good to know. Do you have any opinions on #3438?

Not really, the biggest thing is just the same 'message editing' problem as with the commercial Anthropic CachePoint -- you'd probably see worse cache performance with Nova models + CachePoint than with implicit caching.

I was planning to implement that using proper dataclasses abstracting over everything the big three support, rather than having provider-specific typed dicts. Would love your thoughts on #3126.

Seems reasonable -- the streaming implementations are thankfully pretty straightforward IME, at least for Anthropic & Gemini, so that'll be nice. Have not experimented with OAI file search citations.