Bedrock multimodal embeddings

[jlonge4]

Was thinking of adding the Bedrock multimodal embeddings model to the bedrock_embedding_encoder class. However with its short context length and incredibly specific use case, image / image + caption (128 max input tokens) ....what are your thoughts? @masci @anakin87

[vblagoje]

@jlonge4 we are working on new Claude 3 model support. After the support for these new Claude 3 models has been added - we can take a look. Claude 3 supports multimodal inferencing.

[jlonge4]

@vblagoje Awesome, and right, that's why I was hoping to also leverage the bedrock multimodal embeddings model to retrieve those images and feed them to Claude 3 in a rag pipeline. Great work you guys do! As usual let me know if I can help out, I'm down to create a PR just wanted to pick your brains about where it should live. Thanks!

[vblagoje]

Awesome @jlonge4 if you have time would you please try the branch for Claude 3 deepset-ai/haystack-core-integrations#545 and see if you can break it somehow. There is no multimodal support yet as we might need to retrofit stuff around a bit but text generation should work.

[jlonge4]

@vblagoje hey man sorry it took me a few to find the time but I tried it out and text generation worked just fine!

`from haystack_integrations.components.generators.amazon_bedrock import AmazonBedrockChatGenerator
from haystack.dataclasses import ChatMessage

generator = AmazonBedrockChatGenerator(model="anthropic.claude-3-sonnet-20240229-v1:0", )
messages = [ChatMessage.from_system("You are a helpful assistant that answers questions"), ChatMessage.from_user("What specific model are you?")]

response = generator.run(messages)
print(response)`

Output:

{'replies': [ChatMessage(content="I am an AI model created by Anthropic, but I don't know all the details about what specific model I am. I try not to compare myself directly to other AI models or assistants. What I can say is that I'm a large language model trained to be a general-purpose AI assistant.", role=<ChatRole.ASSISTANT: 'assistant'>, name=None, meta={})]}