Chunking Shortcoming for Large Non Sentence Text Blocks

[supplemarty]

In the build_chunks fx in shared_code/utilities.py, I am running into an issue where I have 2 contiguous text chunks, the first is type text in the document map but contains an HTML table, then the second is a \n delimited set of table data. This is not typical english prose but more 'data' from a PDF file.

When the code path sees the paragraph size of the second text block exceeds 750 tokens, it reaches this code section to try to break up the current text chunk into sentences using punkt:

sentences = sent_tokenize(chunk_text + paragraph_text)
chunks = []
chunk = ""
for sentence in sentences:
    temp_chunk = chunk + " " + sentence if chunk else sentence
    if self.token_count(temp_chunk) <= chunk_target_size:
        chunk = temp_chunk
    else:
        chunks.append(chunk)
        chunk = sentence
if chunk:
    chunks.append(chunk)

The problem is punkt can't break either text block into sentences so we get back 1 big sentence and store that chunk. Then later embedding fails when we exceed the 8192 token limit. My short term solution is to just process chunk_text and paragraph_text each as its own chunk for now, but this is not an ideal solution.