Does LangChain support OpenAI's Batch API?

[Garand0o0]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

''''''

Description

Does LangChain support OpenAI's Batch API?

System Info

langchain

[dosubot[bot]]

Hey @Garand0o0! I'm here to help you with any bugs, questions, or contributions while you wait for a human maintainer. Let me know how I can assist you!

Yes, LangChain supports OpenAI's Batch API. You can configure the batch size and other parameters to use this feature effectively. Here is an example configuration for batch size:

batch_size: int = 20
"""Batch size to use when passing multiple documents to generate."""

For more details, you can refer to the relevant code in the repository ^[1].

To continue talking to Dosu, mention @dosu.

[Garand0o0]

@dosu
hi, is this using OpenAI's Batch API, which reduces costs by 50%?

[dosubot[bot]]

Yes, LangChain's implementation leverages OpenAI's Batch API, which helps in reducing costs by processing embeddings in batches. This approach reduces the number of API calls, thereby taking advantage of the cost-saving benefits of OpenAI's Batch API ^[1].

To continue talking to Dosu, mention @dosu.

[tuhinmallick]

@dosu Can you give a code example how to use langchain batch api functionality ?

[mfsi-suryar]

@dosu
I am also in need of utilizing the OpenAI's Batch API but the batch() method of chains in Langchain is only a wrapper of parallel invoke() and uses the REST endpoints.

[codeananda]

Duplicate question here. Add your upvotes people!