[Feature] Improve catalog lock for paimon

[FangYongs]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently paimon only support hive lock for hive catalog, it should support unified lock for filesystem catalog

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[chenxi0599]

Great！I want this feature, too.

[legendtkl]

+1 for this.

[JingsongLi]

+1 thanks @FangYongs for driving!

[JingsongLi]

Hi, after reviewing #3076

I feel more description should be provided here, otherwise it may lead to incorrect design by contributors.

• What is the overall options API? Do option keys require a lock prefix?
• How does FileSystemCatalog build Lock? How should JdbcLock handle without a jdbc lock context? Do we need to make some modifications to JDBCLock and HiveLock?

I suggest having a clear design before creating subtasks. What do you think? @FangYongs

[FangYongs]

Thanks @JingsongLi , and I completely agree with your opinion! I will create a PIP for this feature and start a discussion thread later.

[sunxiaojian]

@FangYongs @JingsongLi

1. I think that the parameters for LockContext to be open to the public should only be 'option', so there is no need to reference the connection implementation between jdbc and hive in the Filesystem Catalog, which is the future ClientPool

2.I think the parameters required for lock should be distinguished by adding a prefix of "lock." This will not cause any inconvenience to users. However, metastores like jdbc and hive, which come with their own Lock implementation, need to be compatible and do not require redeclaration of lock specific configuration parameters

3. The core issue is the issue of connection. If we need to create a new connection every time and close it after use, it is inevitable to have too many connections, and it may not be possible to seize them during busy times. Therefore, it is recommended to use cache to cache and set the eviction time

If possible, I can refactor according to the above description first.

[rmotapar]

+1 for this