In-context document support for Anthropic and Google models #5130
Conversation
|
Thanks for working on this, but it should not be available only when RAG_API_URL is not set. I am going to work on this soon. My vision for this is adding it to the current list of file upload options: |
|
Ok! I can update the MR to do something similar reusing the Maybe something like this? (with the right icon) On the backend side, the requests could be differentiated by marking RAG files with I'm happy to start working on these changes, or if you'd prefer to handle it, that's fine as well. Let me know what you'd like to do. |
|
@alex-torregrosa thank you that looks good, only small nitpick maybe "Upload to Provider" makes more sense.
Sounds good to me. I'm happy to let you work on it. Note that I was planning on migrating all the file options to every non-agents endpoint, and then start using "ephemeral" agents when certain options/features are selected. The plan is to eventually migrate all backend chat operations to agents, as there is improved performance there and extends capabilities for tool use (which is often assumed as it is for ChatGPT, as is the motivation for this change). I can work on that after your changes, though, as I can see that being iterative over the current scope. |
alex-torregrosa
force-pushed
the
images-to-model
branch
from
a7cc8b0 to
a60c058
Compare
|
Done!
Now the menu will show "Upload to provider" for providers that support more file types than images, and "Upload image" otherwise: Some rough file filtering is also done per-model, similar to the old |
Reused the image handling functions, as the mime type was already being set correctly. For now the upload only happens if RAG is disabled. A better approach would be to re-use the agents menu for deciding context vs RAG
Will be generalized for any file type, not just images
Removed as much references to images as possible
Sets mime type filters for the Google and Anthropic endpoints according to the documents that can be attached in the chat context. Also brings support to upload audio files directly to Gemini.
Added support for attaching documents to claude 3.5
AttachFileMenu is now used for all endpoints, and RAG file uploads are sent with the `file_search` tool. Based on the tool, the upload handler can now select between vectorDB or local files, without need to check for an empty `RAG_API_URL`.
alex-torregrosa
force-pushed
the
images-to-model
branch
from
a60c058 to
99d7bd2
Compare
Summary
Added support for sending full documents in the chat context for Anthropic and Google endpoints.
New available types:
Limitations:
Feel free to suggest any changes or anything else that might be needed.
Change Type
Testing
Tested both the Anthropic & Google endpoints uploading PDF files and requesting a summary. Same with audio files on the Google endpoint.
Checklist