[Bug] thread is blocked when executing show tables from catalog.database via DLF.

[songpcmusic]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

0.9-SNAPSHOT

Compute Engine

StarRocks version 3.3.2

Minimal reproduce step

1. create external catalog default_catalog, do not set paimon.option.client-pool-size, use default value;
2. create database default_db, create large number of tables in this database;
3. execute show tables from default_catalog.default_db in more than one MySQL connection at the same time;
4. show tables from default_catalog.default_db can not get result.

What doesn't meet your expectations?

1. could not get show tables from default_catalog.default_db result when more than one MySQL connection;
2. and all metadata operations will execute fail;

Anything else?

1. jstack
   "starrocks-mysql-nio-pool-3" #18223 daemon prio=5 os_prio=0 cpu=9519.40ms elapsed=7381.20s tid=0x00007f112800e000 nid=0x48d2 in Object.wait() [0x00007f10f6414000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait([email protected]/Native Method) - waiting on <no object reference available> at org.apache.paimon.client.ClientPool$ClientPoolImpl.get(ClientPool.java:189) - waiting to re-lock in wait() <0x00000001ca8b36d0> (a java.lang.Object) at org.apache.paimon.client.ClientPool$ClientPoolImpl.run(ClientPool.java:83) at org.apache.paimon.client.ClientPool$ClientPoolImpl.run(ClientPool.java:78) at org.apache.paimon.hive.pool.CachedClientPool.run(CachedClientPool.java:133) at org.apache.paimon.hive.HiveCatalog.tableExists(HiveCatalog.java:407) at org.apache.paimon.hive.HiveCatalog.lambda$null$6(HiveCatalog.java:387) at org.apache.paimon.hive.HiveCatalog$$Lambda$1012/0x0000000840e2d840.test(Unknown Source) at java.util.stream.ReferencePipeline$2$1.accept([email protected]/ReferencePipeline.java:176) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining([email protected]/ArrayList.java:1655) at java.util.stream.AbstractPipeline.copyInto([email protected]/AbstractPipeline.java:484) at java.util.stream.AbstractPipeline.wrapAndCopyInto([email protected]/AbstractPipeline.java:474) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential([email protected]/ReduceOps.java:913) at java.util.stream.AbstractPipeline.evaluate([email protected]/AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect([email protected]/ReferencePipeline.java:578) at org.apache.paimon.hive.HiveCatalog.lambda$listTablesImpl$7(HiveCatalog.java:389) at org.apache.paimon.hive.HiveCatalog$$Lambda$1007/0x0000000840e2a440.run(Unknown Source) at org.apache.paimon.client.ClientPool$ClientPoolImpl.run(ClientPool.java:85) at org.apache.paimon.client.ClientPool$ClientPoolImpl.run(ClientPool.java:78) at org.apache.paimon.hive.pool.CachedClientPool.run(CachedClientPool.java:133) at org.apache.paimon.hive.HiveCatalog.listTablesImpl(HiveCatalog.java:380) at org.apache.paimon.catalog.AbstractCatalog.listTables(AbstractCatalog.java:202) at org.apache.paimon.catalog.DelegateCatalog.listTables(DelegateCatalog.java:105) at com.starrocks.connector.paimon.PaimonMetadata.listTableNames(PaimonMetadata.java:105) at com.starrocks.connector.CatalogConnectorMetadata.listTableNames(CatalogConnectorMetadata.java:114) at com.starrocks.server.MetadataMgr.listTableNames(MetadataMgr.java:259) at com.starrocks.qe.ShowExecutor$ShowExecutorVisitor.visitShowTableStatement(ShowExecutor.java:478) at com.starrocks.qe.ShowExecutor$ShowExecutorVisitor.visitShowTableStatement(ShowExecutor.java:288) at com.starrocks.sql.ast.ShowTableStmt.accept(ShowTableStmt.java:131) at com.starrocks.sql.ast.AstVisitor.visit(AstVisitor.java:71) at com.starrocks.qe.ShowExecutor.execute(ShowExecutor.java:285) at com.starrocks.qe.StmtExecutor.handleShow(StmtExecutor.java:1679) at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:692) at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:409) at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:604) at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:938) at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) at com.starrocks.mysql.nio.ReadListener$$Lambda$481/0x000000084083bc40.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628) at java.lang.Thread.run([email protected]/Thread.java:829)
2. souce code, org.apache.paimon.hive.HiveCatalog#listTablesImpl

protected List<String> listTablesImpl(String databaseName) {
        try {
            return clients.run(
                    client ->
                            client.getAllTables(databaseName).stream()
                                    .filter(
                                            tableName -> {
                                                Identifier identifier =
                                                        new Identifier(databaseName, tableName);
                                                return tableExists(identifier);
                                            })
                                    .collect(Collectors.toList()));
        } catch (TException e) {
            throw new RuntimeException("Failed to list all tables in database " + databaseName, e);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException("Interrupted in call to listTables " + databaseName, e);
        }
    }

3. cause,

• clients.run execute getAllTables, get one client, and filter iterate over each table;
• every tableExists operation need one more client. if only one MySQL connection, default value 2 clients can execute success;
• but if one more MySQL connection at the same time, this connection get one client, the other connection can not get one more client. both two threads will all wait at lock.

[mxdzs0612]

Seems can be fixed by #4468.