Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
chore(deps): Bump unstructured[local-inference] from 0.10.14 to 0.10.…
…15 in /requirements (#242) Bumps [unstructured[local-inference]](https://github.com/Unstructured-IO/unstructured) from 0.10.14 to 0.10.15. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/Unstructured-IO/unstructured/releases">unstructured[local-inference]'s releases</a>.</em></p> <blockquote> <h2>0.10.15</h2> <h3>Enhancements</h3> <ul> <li><strong>Suport for better element categories from the next-generation image-to-text model ("chipper").</strong>. Previously, not all of the classifications from Chipper were being mapped to proper <code>unstructured</code> element categories so the consumer of the library would see many <code>UncategorizedText</code> elements. This fixes the issue, improving the granularity of the element categories outputs for better downstream processing and chunking. The mapping update is: <ul> <li>"Threading": <code>NarrativeText</code></li> <li>"Form": <code>NarrativeText</code></li> <li>"Field-Name": <code>Title</code></li> <li>"Value": <code>NarrativeText</code></li> <li>"Link": <code>NarrativeText</code></li> <li>"Headline": <code>Title</code> (with <code>category_depth=1</code>)</li> <li>"Subheadline": <code>Title</code> (with <code>category_depth=2</code>)</li> <li>"Abstract": <code>NarrativeText</code></li> </ul> </li> <li><strong>Better ListItem grouping for PDF's (fast strategy).</strong> The <code>partition_pdf</code> with <code>fast</code> strategy previously broke down some numbered list item lines as separate elements. This enhancement leverages the x,y coordinates and bbox sizes to help decide whether the following chunk of text is a continuation of the immediate previous detected ListItem element or not, and not detect it as its own non-ListItem element.</li> <li><strong>Fall back to text-based classification for uncategorized Layout elements for Images and PDF's</strong>. Improves element classification by running existing text-based rules on previously UncategorizedText elements</li> <li><strong>Adds table partitioning for Partitioning for many doc types including: .html, .epub., .md, .rst, .odt, and .msg.</strong> At the core of this change is the .html partition functionality, which is leveraged by the other effected doc types. This impacts many scenarios where <code>Table</code> Elements are now propery extracted.</li> <li><strong>Create and add <code>add_chunking_strategy</code> decorator to partition functions.</strong> Previously, users were responsible for their own chunking after partitioning elements, often required for downstream applications. Now, individual elements may be combined into right-sized chunks where min and max character size may be specified if <code>chunking_strategy=by_title</code>. Relevant elements are grouped together for better downstream results. This enables users immediately use partitioned results effectively in downstream applications (e.g. RAG architecture apps) without any additional post-processing.</li> <li><strong>Adds <code>languages</code> as an input parameter and marks <code>ocr_languages</code> kwarg for deprecation in pdf, image, and auto partitioning functions.</strong> Previously, language information was only being used for Tesseract OCR for image-based documents and was in a Tesseract specific string format, but by refactoring into a list of standard language codes independent of Tesseract, the <code>unstructured</code> library will better support <code>languages</code> for other non-image pipelines and/or support for other OCR engines.</li> <li><strong>Removes <code>UNSTRUCTURED_LANGUAGE</code> env var usage and replaces <code>language</code> with <code>languages</code> as an input parameter to unstructured-partition-text_type functions.</strong> The previous parameter/input setup was not user-friendly or scalable to the variety of elements being processed. By refactoring the inputted language information into a list of standard language codes, we can support future applications of the element language such as detection, metadata, and multi-language elements. Now, to skip English specific checks, set the <code>languages</code> parameter to any non-English language(s).</li> <li><strong>Adds <code>xlsx</code> and <code>xls</code> filetype extensions to the <code>skip_infer_table_types</code> default list in <code>partition</code>.</strong> By adding these file types to the input parameter these files should not go through table extraction. Users can still specify if they would like to extract tables from these filetypes, but will have to set the <code>skip_infer_table_types</code> to exclude the desired filetype extension. This avoids mis-representing complex spreadsheets where there may be multiple sub-tables and other content.</li> <li><strong>Better debug output related to sentence counting internals</strong>. Clarify message when sentence is not counted toward sentence count because there aren't enough words, relevant for developers focused on <code>unstructured</code>s NLP internals.</li> <li><strong>Faster ocr_only speed for partitioning PDF and images.</strong> Use <code>unstructured_pytesseract.run_and_get_multiple_output</code> function to reduce the number of calls to <code>tesseract</code> by half when partitioning pdf or image with <code>tesseract</code></li> <li><strong>Adds data source properties to fsspec connectors</strong> These properties (date_created, date_modified, version, source_url, record_locator) are written to element metadata during ingest, mapping elements to information about the document source from which they derive. This functionality enables downstream applications to reveal source document applications, e.g. a link to a GDrive doc, Salesforce record, etc.</li> <li><strong>Add delta table destination connector</strong> New delta table destination connector added to ingest CLI. Users may now use <code>unstructured-ingest</code> to write partitioned data from over 20 data sources (so far) to a Delta Table.</li> <li><strong>Rename to Source and Destination Connectors in the Documentation.</strong> Maintain naming consistency between Connectors codebase and documentation with the first addition to a destination connector.</li> <li><strong>Non-HTML text files now return unstructured-elements as opposed to HTML-elements.</strong> Previously the text based files that went through <code>partition_html</code> would return HTML-elements but now we preserve the format from the input using <code>source_format</code> argument in the partition call.</li> <li><strong>Adds <code>PaddleOCR</code> as an optional alternative to <code>Tesseract</code></strong> for OCR in processing of PDF or Image files, it is installable via the <code>makefile</code> command <code>install-paddleocr</code>. For experimental purposes only.</li> <li><strong>Bump unstructured-inference</strong> to 0.5.28. This version bump markedly improves the output of table data, rendered as <code>metadata.text_as_html</code> in an element. These changes include: <ul> <li>add env variable <code>ENTIRE_PAGE_OCR</code> to specify using paddle or tesseract on entire page OCR</li> <li>table structure detection now pads the input image by 25 pixels in all 4 directions to improve its recall (0.5.27)</li> <li>support paddle with both cpu and gpu and assumed it is pre-installed (0.5.26)</li> <li>fix a bug where <code>cells_to_html</code> doesn't handle cells spanning multiple rows properly (0.5.25)</li> <li>remove <code>cv2</code> preprocessing step before OCR step in table transformer (0.5.24)</li> </ul> </li> </ul> <h3>Features</h3> <ul> <li><strong>Adds element metadata via <code>category_depth</code> with default value None</strong>. <ul> <li>This additional metadata is useful for vectordb/LLM, chunking strategies, and retrieval applications.</li> </ul> </li> <li><strong>Adds a naive hierarchy for elements via a <code>parent_id</code> on the element's metadata</strong> <ul> <li>Users will now have more metadata for implementing vectordb/LLM chunking strategies. For example, text elements could be queried by their preceding title element.</li> <li>Title elements created from HTML headings will properly nest</li> </ul> </li> </ul> <h3>Fixes</h3> <ul> <li><strong><code>add_pytesseract_bboxes_to_elements</code> no longer returns <code>nan</code> values</strong>. The function logic is now broken into new methods <code>_get_element_box</code> and <code>convert_multiple_coordinates_to_new_system</code></li> <li><strong>Selecting a different model wasn't being respected when calling <code>partition_image</code>.</strong> Problem: <code>partition_pdf</code> allows for passing a <code>model_name</code> parameter. Given the similarity between the image and PDF pipelines, the expected behavior is that <code>partition_image</code> should support the same parameter, but <code>partition_image</code> was unintentionally not passing along its <code>kwargs</code>. This was corrected by adding the kwargs to the downstream call.</li> <li><strong>Fixes a chunking issue via dropping the field "coordinates".</strong> Problem: chunk_by_title function was chunking each element to its own individual chunk while it needed to group elements into a fewer number of chunks. We've discovered that this happens due to a metadata matching logic in chunk_by_title function, and discovered that elements with different metadata can't be put into the same chunk. At the same time, any element with "coordinates" essentially had different metadata than other elements, due each element locating in different places and having different coordinates. Fix: That is why we have included the key "coordinates" inside a list of excluded metadata keys, while doing this "metadata_matches" comparision. Importance: This change is crucial to be able to chunk by title for documents which include "coordinates" metadata in their elements.</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/Unstructured-IO/unstructured/blob/main/CHANGELOG.md">unstructured[local-inference]'s changelog</a>.</em></p> <blockquote> <h2>0.10.15</h2> <h3>Enhancements</h3> <ul> <li><strong>Suport for better element categories from the next-generation image-to-text model ("chipper").</strong>. Previously, not all of the classifications from Chipper were being mapped to proper <code>unstructured</code> element categories so the consumer of the library would see many <code>UncategorizedText</code> elements. This fixes the issue, improving the granularity of the element categories outputs for better downstream processing and chunking. The mapping update is: <ul> <li>"Threading": <code>NarrativeText</code></li> <li>"Form": <code>NarrativeText</code></li> <li>"Field-Name": <code>Title</code></li> <li>"Value": <code>NarrativeText</code></li> <li>"Link": <code>NarrativeText</code></li> <li>"Headline": <code>Title</code> (with <code>category_depth=1</code>)</li> <li>"Subheadline": <code>Title</code> (with <code>category_depth=2</code>)</li> <li>"Abstract": <code>NarrativeText</code></li> </ul> </li> <li><strong>Better ListItem grouping for PDF's (fast strategy).</strong> The <code>partition_pdf</code> with <code>fast</code> strategy previously broke down some numbered list item lines as separate elements. This enhancement leverages the x,y coordinates and bbox sizes to help decide whether the following chunk of text is a continuation of the immediate previous detected ListItem element or not, and not detect it as its own non-ListItem element.</li> <li><strong>Fall back to text-based classification for uncategorized Layout elements for Images and PDF's</strong>. Improves element classification by running existing text-based rules on previously UncategorizedText elements</li> <li><strong>Adds table partitioning for Partitioning for many doc types including: .html, .epub., .md, .rst, .odt, and .msg.</strong> At the core of this change is the .html partition functionality, which is leveraged by the other effected doc types. This impacts many scenarios where <code>Table</code> Elements are now propery extracted.</li> <li><strong>Create and add <code>add_chunking_strategy</code> decorator to partition functions.</strong> Previously, users were responsible for their own chunking after partitioning elements, often required for downstream applications. Now, individual elements may be combined into right-sized chunks where min and max character size may be specified if <code>chunking_strategy=by_title</code>. Relevant elements are grouped together for better downstream results. This enables users immediately use partitioned results effectively in downstream applications (e.g. RAG architecture apps) without any additional post-processing.</li> <li><strong>Adds <code>languages</code> as an input parameter and marks <code>ocr_languages</code> kwarg for deprecation in pdf, image, and auto partitioning functions.</strong> Previously, language information was only being used for Tesseract OCR for image-based documents and was in a Tesseract specific string format, but by refactoring into a list of standard language codes independent of Tesseract, the <code>unstructured</code> library will better support <code>languages</code> for other non-image pipelines and/or support for other OCR engines.</li> <li><strong>Removes <code>UNSTRUCTURED_LANGUAGE</code> env var usage and replaces <code>language</code> with <code>languages</code> as an input parameter to unstructured-partition-text_type functions.</strong> The previous parameter/input setup was not user-friendly or scalable to the variety of elements being processed. By refactoring the inputted language information into a list of standard language codes, we can support future applications of the element language such as detection, metadata, and multi-language elements. Now, to skip English specific checks, set the <code>languages</code> parameter to any non-English language(s).</li> <li><strong>Adds <code>xlsx</code> and <code>xls</code> filetype extensions to the <code>skip_infer_table_types</code> default list in <code>partition</code>.</strong> By adding these file types to the input parameter these files should not go through table extraction. Users can still specify if they would like to extract tables from these filetypes, but will have to set the <code>skip_infer_table_types</code> to exclude the desired filetype extension. This avoids mis-representing complex spreadsheets where there may be multiple sub-tables and other content.</li> <li><strong>Better debug output related to sentence counting internals</strong>. Clarify message when sentence is not counted toward sentence count because there aren't enough words, relevant for developers focused on <code>unstructured</code>s NLP internals.</li> <li><strong>Faster ocr_only speed for partitioning PDF and images.</strong> Use <code>unstructured_pytesseract.run_and_get_multiple_output</code> function to reduce the number of calls to <code>tesseract</code> by half when partitioning pdf or image with <code>tesseract</code></li> <li><strong>Adds data source properties to fsspec connectors</strong> These properties (date_created, date_modified, version, source_url, record_locator) are written to element metadata during ingest, mapping elements to information about the document source from which they derive. This functionality enables downstream applications to reveal source document applications, e.g. a link to a GDrive doc, Salesforce record, etc.</li> <li><strong>Add delta table destination connector</strong> New delta table destination connector added to ingest CLI. Users may now use <code>unstructured-ingest</code> to write partitioned data from over 20 data sources (so far) to a Delta Table.</li> <li><strong>Rename to Source and Destination Connectors in the Documentation.</strong> Maintain naming consistency between Connectors codebase and documentation with the first addition to a destination connector.</li> <li><strong>Non-HTML text files now return unstructured-elements as opposed to HTML-elements.</strong> Previously the text based files that went through <code>partition_html</code> would return HTML-elements but now we preserve the format from the input using <code>source_format</code> argument in the partition call.</li> <li><strong>Adds <code>PaddleOCR</code> as an optional alternative to <code>Tesseract</code></strong> for OCR in processing of PDF or Image files, it is installable via the <code>makefile</code> command <code>install-paddleocr</code>. For experimental purposes only.</li> <li><strong>Bump unstructured-inference</strong> to 0.5.28. This version bump markedly improves the output of table data, rendered as <code>metadata.text_as_html</code> in an element. These changes include: <ul> <li>add env variable <code>ENTIRE_PAGE_OCR</code> to specify using paddle or tesseract on entire page OCR</li> <li>table structure detection now pads the input image by 25 pixels in all 4 directions to improve its recall (0.5.27)</li> <li>support paddle with both cpu and gpu and assumed it is pre-installed (0.5.26)</li> <li>fix a bug where <code>cells_to_html</code> doesn't handle cells spanning multiple rows properly (0.5.25)</li> <li>remove <code>cv2</code> preprocessing step before OCR step in table transformer (0.5.24)</li> </ul> </li> </ul> <h3>Features</h3> <ul> <li><strong>Adds element metadata via <code>category_depth</code> with default value None</strong>. <ul> <li>This additional metadata is useful for vectordb/LLM, chunking strategies, and retrieval applications.</li> </ul> </li> <li><strong>Adds a naive hierarchy for elements via a <code>parent_id</code> on the element's metadata</strong> <ul> <li>Users will now have more metadata for implementing vectordb/LLM chunking strategies. For example, text elements could be queried by their preceding title element.</li> <li>Title elements created from HTML headings will properly nest</li> </ul> </li> </ul> <h3>Fixes</h3> <ul> <li><strong><code>add_pytesseract_bboxes_to_elements</code> no longer returns <code>nan</code> values</strong>. The function logic is now broken into new methods <code>_get_element_box</code> and <code>convert_multiple_coordinates_to_new_system</code></li> <li><strong>Selecting a different model wasn't being respected when calling <code>partition_image</code>.</strong> Problem: <code>partition_pdf</code> allows for passing a <code>model_name</code> parameter. Given the similarity between the image and PDF pipelines, the expected behavior is that <code>partition_image</code> should support the same parameter, but <code>partition_image</code> was unintentionally not passing along its <code>kwargs</code>. This was corrected by adding the kwargs to the downstream call.</li> <li><strong>Fixes a chunking issue via dropping the field "coordinates".</strong> Problem: chunk_by_title function was chunking each element to its own individual chunk while it needed to group elements into a fewer number of chunks. We've discovered that this happens due to a metadata matching logic in chunk_by_title function, and discovered that elements with different metadata can't be put into the same chunk. At the same time, any element with "coordinates" essentially had different metadata than other elements, due each element locating in different places and having different coordinates. Fix: That is why we have included the key "coordinates" inside a list of excluded metadata keys, while doing this "metadata_matches" comparision. Importance: This change is crucial to be able to chunk by title for documents which include "coordinates" metadata in their elements.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/Unstructured-IO/unstructured/commit/b534b2a6cdd91f2a6380b8e1097de28e141a0d8e"><code>b534b2a</code></a> Chore: bump inference package version to 0.5.28 and new release (<a href="https://redirect.github.com/Unstructured-IO/unstructured/issues/1355">#1355</a>)</li> <li><a href="https://github.com/Unstructured-IO/unstructured/commit/09a0958f900a6748c217c9f022ca90b4ab01b3a5"><code>09a0958</code></a> Feat: CORE-1269 - Install paddlepaddle wheel dependent on arch, supporting aa...</li> <li><a href="https://github.com/Unstructured-IO/unstructured/commit/36d026cb1bb009a275c3afb6a79bd7237c762027"><code>36d026c</code></a> chore: update CHANGELOG.md bullets (<a href="https://redirect.github.com/Unstructured-IO/unstructured/issues/1436">#1436</a>)</li> <li><a href="https://github.com/Unstructured-IO/unstructured/commit/6187dc09768df825920dca0e323005712aad05d2"><code>6187dc0</code></a> update links in integrations.rst (<a href="https://redirect.github.com/Unstructured-IO/unstructured/issues/1418">#1418</a>)</li> <li><a href="https://github.com/Unstructured-IO/unstructured/commit/333558494e6695717b73421cf9fea1a3285925ef"><code>3335584</code></a> roman/delta lake dest connector (<a href="https://redirect.github.com/Unstructured-IO/unstructured/issues/1385">#1385</a>)</li> <li><a href="https://github.com/Unstructured-IO/unstructured/commit/98d3541909f64290b5efb65a226fc3ee8a7cc5ee"><code>98d3541</code></a> Update CHANGELOG.md (<a href="https://redirect.github.com/Unstructured-IO/unstructured/issues/1435">#1435</a>)</li> <li><a href="https://github.com/Unstructured-IO/unstructured/commit/de4d496fcf64cfadfcdc4ab065c106287eb48637"><code>de4d496</code></a> Fix bbox coordinates for ocr_only strategy (<a href="https://redirect.github.com/Unstructured-IO/unstructured/issues/1325">#1325</a>)</li> <li><a href="https://github.com/Unstructured-IO/unstructured/commit/0d61c9848170b8db090b121b97e3822dbeff4eab"><code>0d61c98</code></a> fix: Pass partition_image kwargs downstream (<a href="https://redirect.github.com/Unstructured-IO/unstructured/issues/1426">#1426</a>)</li> <li><a href="https://github.com/Unstructured-IO/unstructured/commit/fe11ab4235ad2b2bc8328a036b4da33b7392f8fb"><code>fe11ab4</code></a> feat: improved mapping for missing chipper elements (<a href="https://redirect.github.com/Unstructured-IO/unstructured/issues/1431">#1431</a>)</li> <li><a href="https://github.com/Unstructured-IO/unstructured/commit/50db2abd9f6f0eadd456a4b5026b4ff0dbdc5d75"><code>50db2ab</code></a> fix: updating element types (<a href="https://redirect.github.com/Unstructured-IO/unstructured/issues/1394">#1394</a>)</li> <li>Additional commits viewable in <a href="https://github.com/Unstructured-IO/unstructured/compare/0.10.14...0.10.15">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Austin Walker <[email protected]>
- Loading branch information
3 people
authored
Sep 18, 2023
1 parent
6923a24
commit af24905