[core] Introduce Object Table to manage unstructured files

[JingsongLi]

Purpose

We plan to introduce an Object Table to provide metadata indexes for unstructured data objects in the specified Object Storage storage directory. Object tables allow users to analyze unstructured data in Object Storage:

1. Use Python API to manipulate these unstructured data, such as converting images to PDF format.
2. Model functions can also be used to perform inference, and then the results of these operations can be concatenated with other structured data in the Catalog.

The object table is managed by Catalog and can also have access permissions and the ability to manage blood relations.

The schema for Object Table is:

    RowType SCHEMA =
            RowType.builder()
                    .field("path", DataTypes.STRING().notNull())
                    .field("name", DataTypes.STRING().notNull())
                    .field("length", DataTypes.BIGINT().notNull())
                    .field("mtime", DataTypes.TIMESTAMP_LTZ_MILLIS())
                    .field("atime", DataTypes.TIMESTAMP_LTZ_MILLIS())
                    .field("owner", DataTypes.STRING().nullable())
                    .field("generation", DataTypes.INT().nullable())
                    .field("content_type", DataTypes.STRING().nullable())
                    .field("storage_class", DataTypes.STRING().nullable())
                    .field("md5_hash", DataTypes.STRING().nullable())
                    .field("metadata_mtime", DataTypes.TIMESTAMP_LTZ_MILLIS().nullable())
                    .field("metadata", DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING()))
                    .build()
                    .notNull();

The SQL usage is:

-- Create Object Table

CREATE TABLE `my_object_table` WITH (
  'type' = 'object-table',
  'object-location' = 'oss://my_bucket/my_location' 
)

-- Refresh Object Table

CALL sys.refresh_object_table('mydb.my_object_table');

-- Query Object Table

SELECT * FROM `my_object_table`;

-- Query Object Table with Time Travel

SELECT * FROM `my_object_table` /*+ OPTIONS('scan.snapshot-id' = '1') */;

Tests

ObjectTableITCase

API and Format

Documentation

[wwj6591812]

I think getLatestAccessTime may better?

[JingsongLi]

Same to getModificationTime.

[wwj6591812]

Add this to doc?

[wwj6591812]

Object table does not support Commit.

[wwj6591812]

Check options.objectLocation() is null?

[wwj6591812]

”We plan to introduce an Object Table to provide metadata indexes for unstructured data objects in the specified Object Storage storage directory.“

Hi, jingsong.
After see the pip, no relevant information was found about Paimon supporting unstructured data storage.
Could you describe Paimon’s future plans in the unstructured data storage? Our company is currently exploring how to use Paimon for unstructured analysis.

[JingsongLi]

”We plan to introduce an Object Table to provide metadata indexes for unstructured data objects in the specified Object Storage storage directory.“

Hi, jingsong. After see the pip, no relevant information was found about Paimon supporting unstructured data storage. Could you describe Paimon’s future plans in the unstructured data storage? Our company is currently exploring how to use Paimon for unstructured analysis.

Unstructured data is definitely not stored in Paimon for the most cases, so what we want to do here is simply map these files so that users can process the metadata of unstructured files and see if they need to read these files as needed.

[yanbinyang]

+1