Max Chunk size created for the final chunk returned from the SementicChunker.split_document()

[akash97715]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

text_splitter = SemanticChunker(
    OpenAIEmbeddings(), breakpoint_threshold_type="percentile"
)

docs = text_splitter.create_documents([state_of_the_union])
print(docs[0].page_content)

Description

I am using the above sementic chunking for langchain to create the chunk. I just want to undesrtand what can be the max size of chunk returned from this function. Is it greater than 8k token?? if yes then i need to do some postprocessing to hit Azure embdding endpoint. This length piece is missing in current document.

System Info

Langchain latest

[akash97715]

@dosubot please respond

[akash97715]

@dosu please response