[schema] Support database path for specified database in catalog

[FangYongs]

Purpose

Linked issue: close #2493

Currently there will be FileIO when a catalog is created. We would like to support path for database and paimon need to get fileio for each database when it is created, dropped and tables are created, dropped for it. This PR aims to introduce DatabaseFileIOProvider in catalog to get fileio for given database name.

Tests

Exist tests can cover this feature.

API and Format

no

Documentation

no

[JingsongLi]

Hi @FangYongs
Does this come from real business needs? In my previous understanding, this situation is usually solved by the FileSystem of HDFS. (Or Flink FileSystem)

[FangYongs]

Hi @FangYongs Does this come from real business needs? In my previous understanding, this situation is usually solved by the FileSystem of HDFS. (Or Flink FileSystem)

@JingsongLi Yes. Currently in our production environment, we always support one catalog for hive, hudi, iceberg or paimon, so they can easily read/write tables use paimon.databaseName.tableName. We have a meta service in company to manage all databases and tables for each catalog. When users create a database, they need to give a path of hdfs/s3 for it and then create tables in the database. So we need to support configuring warehosue path in database level.

[JingsongLi]

Hi @FangYongs Does this come from real business needs? In my previous understanding, this situation is usually solved by the FileSystem of HDFS. (Or Flink FileSystem)

@JingsongLi Yes. Currently in our production environment, we always support one catalog for hive, hudi, iceberg or paimon, so they can easily read/write tables use paimon.databaseName.tableName. We have a meta service in company to manage all databases and tables for each catalog. When users create a database, they need to give a path of hdfs/s3 for it and then create tables in the database. So we need to support configuring warehosue path in database level.

Can Hadoop FileSystem solve this? #2504

[FangYongs]

@JingsongLi It seems not, HadoopFileIO is for catalog too. As mentioned above, we have a meta service for all catalogs, so we create a paimon meta catalog based on it, and the catalog stores the database and path after users created it. When users create a table, it needs to find the database and its path from the paimon meta catalog and create FileIO for the table to store schema and data, that's why I would like to introduce the interface DatabaseFileIOProvider in AbstractCatalog to get FileIO for specified database.

What do you think?

[JingsongLi]

@JingsongLi It seems not, HadoopFileIO is for catalog too. As mentioned above, we have a meta service for all catalogs, so we create a paimon meta catalog based on it, and the catalog stores the database and path after users created it. When users create a table, it needs to find the database and its path from the paimon meta catalog and create FileIO for the table to store schema and data, that's why I would like to introduce the interface DatabaseFileIOProvider in AbstractCatalog to get FileIO for specified database.

What do you think?

Can you list what FileSystems need to be supported?

[JingsongLi]

For example, Hadoop FileSystem supports viewfs, there can be multiple underly filesystems.

[FangYongs]

@JingsongLi Local file system, hdfs or s3 which have beed supported in paimon

[FangYongs]

For example, Hadoop FileSystem supports viewfs, there can be multiple underly filesystems.

We may configure different hdfs cluster for different databases in the paimon catalog, do you mean we can get different filesystem from one FileIO?

[JingsongLi]

For example, Hadoop FileSystem supports viewfs, there can be multiple underly filesystems.

We may configure different hdfs cluster for different databases in the paimon catalog, do you mean we can get different filesystem from one FileIO?

Yes

[JingsongLi]

From Paimon Level, you can do this too. Just like current HadoopFileIO, there is a Map<Pair<String, String>, FileSystem> fsMap, which means scheme + authority => FileSystem.

[FangYongs]

@JingsongLi Get it. Then I think we can add minor updates in paimon to support different database paths for catalog, I will update this PR

[FangYongs]

@JingsongLi What do you think of this PR now? We can implement customized catalog and get warehouse path with specified database name

[FangYongs]

@JingsongLi Can you help to review this PR again? THX

[JingsongLi]

+1