[Bedrock] Implement prompt caching for supported models

[richardburleigh]

Describe the feature

Anthropic have introduced Prompt Caching for Claude which allows significantly faster inference. Adding support for this would be of significant value to customers.

Use Case

When using long system prompts or multi-agent applications, it can sometimes take up to 30 seconds or more for a response to be received by the customer. Prompt caching is an elegant and easy to implement means of reducing this significantly.

Proposed Solution

Add support for Prompt Caching.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

Latest

Environment details (OS name and version, etc.)

All

[tim-finnigan]

Thanks for the feature request. The Bedrock service team would need to implement support for this, as they own and maintain the underlying APIs (https://docs.aws.amazon.com/bedrock/latest/APIReference/API_Operations.html).

Upon searching internally, I found that there was already an internal feature request for the Bedrock team to track this. I've added your +1 and use case to the tracking item, and if there's any additional info you want us to pass along to the Bedrock team please let us know. I'm going to close the issue since this won't be tracked on the SDK side. But any updates to service APIs like this would be released to the SDKs. Going forward you can refer to the blog and CHANGELOG for updates.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[juan-abia]

Any news on this? It'd be awesome if we could track it

[sammcj]

Any updates on this? This would reduce Bedrock based LLM costs by up to 90%.

[devtanna]

Following up here and commenting for updates. This would indeed be helpful for us

[sammcj]

I'm meeting with a bunch of AWS SAs in the GenAI space tomorrow, I'm raising the lack of prompt caching as a detractor from Bedrock with clients and will reference this issue.

I realise this is really a bedrock issue, but given bedrock is closed source and doesn't have a public issue tracker I'm not sure where folks can track and discuss it.

[FarazAhmedSid]

Is there any tentative date for the release of this feature?

[sammcj]

Folks you're going to need to get onto your Amazon account reps / local SAs to push for this in Bedrock.

[mpbrenlla]

Any updates on this?

[ajinkyaT]

Waiting for updates!

[carl-krikorian]

This would be very helpful if implemented, any updates?

[skapoor-coatue]

Following

[ShivamSphn]

• more caching time like openai

[JavierArredondo]

Following

[emorgan-korrobio]

also following

[sammcj]

Press the follow button and give it a +1 rather that writing "following" which spans everyone with an email.

[marcelslum]

following

[tomaszdudek7]

It has been released right now on re:Invent 2024. :)

[ranman]

https://aws.amazon.com/blogs/aws/reduce-costs-and-latency-with-amazon-bedrock-intelligent-prompt-routing-and-prompt-caching-preview/