[BUG] Managed Identity token acquisition thread will be blocked when the common pool ForkJoinPool exhausted

[moarychan]

Describe the bug
When using Managed Identity scenario, if you're using credential from DefaultAzureCredentialBuilder without a custom ExecutorService or directly using ManagedIdentityCredentialBiulder, you should be aware that the token acquisition thread may not execute in a timely manner.

Due to Managed Identity credential does not support the custom ExecutorService, and it will run the method CompletableFuture.supplyAsync(supplier), see from AbstractApplicationBase#executeRequest

Then the token acquisition thread will use a static common pool ForkJoinPool.commonPool(), see from CompletableFuture#supplyAsync

A more specific case is using Spring Boot + Spring Cloud Azure Starter JDBC MySQL + JPA.
Lots of the user's business processing threads are executed in the common pool java.util.concurrent.ForkJoinPool#common, and they begin to need to connect to the MySQL database. At this time, because the JDBC authentication plugin is available, then it's called to get access token as password, it will use the Managed Identity Credential, each token request thread will not be executed immediately, and the business thread will be stagnant.

Exception or Stack Trace
The token acquisition thread is not started, not found any stack trace within msal4j, stopped here:

"xxx.mysql.database.azure.com/dev_data2 connection adder" #2873 daemon prio=5 os_prio=0 tid=0x00007f3b8801f000 nid=0x200202 waiting on condition [0x00007efa508ae000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006c7c1e760> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:125)
	at reactor.core.publisher.Mono.block(Mono.java:1738)
	at com.azure.identity.extensions.implementation.template.AzureAuthenticationTemplate.getTokenAsPassword(AzureAuthenticationTemplate.java:101)
	at com.azure.identity.extensions.jdbc.mysql.AzureMysqlAuthenticationPlugin.nextAuthenticationStep(AzureMysqlAuthenticationPlugin.java:88)
	at com.azure.identity.extensions.jdbc.mysql.AzureMysqlAuthenticationPlugin.nextAuthenticationStep(AzureMysqlAuthenticationPlugin.java:20)
	at com.mysql.cj.protocol.a.NativeAuthenticationProvider.proceedHandshakeWithPluggableAuthentication(NativeAuthenticationProvider.java:446)
	at com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:215)
	at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1428)
	at com.mysql.cj.NativeSession.connect(NativeSession.java:133)
	at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:935)
	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:805)
	- locked <0x00000006c7b80690> (a com.mysql.cj.jdbc.ConnectionImpl)
	at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:438)
	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:189)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476)
	at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71)
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:726)
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

   Locked ownable synchronizers:
	- <0x00000006c432f508> (a java.util.concurrent.ThreadPoolExecutor$Worker)

and there are lots of threads named started with ForkJoinPool.commonPool-worker-, they occupy common pool java.util.concurrent.ForkJoinPool#common resources.

To Reproduce
Run this reproducer under Azure Managed Identity enabled environment.

Code Snippet
N/A

Expected behavior
Managed Identity credential supports custom ExecutorService, better to use a custom thread pool by default when no executor service provided.

Screenshots
N/A

Setup (please complete the following information):

• OS: [e.g. iOS] Azure Virtual Machine Windows 11
• IDE: [e.g. IntelliJ] Intellij Idea
• Library/Libraries: Spring Boot 3.3.2 + Spring Cloud Azure 5.19.0
• Java version: [e.g. 8] Java 17

Additional context
Related issues:

• [FEATURE REQ] Support setAuthorityHost with CredentialBuilderBase #8626
• [Azure Identity] Deadlock when using common ForkJoinPool #22687.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[github-actions]

@billwert @g2vinay

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[moarychan]

Just tested, please see the the proven workaround.

[moarychan]

Hi @g2vinay , I have following questions, could you please help explain them to us? CC @saragluna

• Why does only default credential builder support custom executor service, while others like the managed identity biulder and the CLI credential biulder do not?
• Why does the managed identity credential not provide the sync acquisition token method instead the default async method with block calling?
• Why does the managed identity credential not provide the caching ability like the cachedToken of UsernamePasswordCredential?

[billwert]

@moarychan Hello! Thanks for investigation.

• Why does only default credential builder support custom executor service, while others like the managed identity biulder and the CLI credential biulder do not?

The API difference is something we can think about. With that said it is my expectation that the MI client being used is using the SharedExecutorService from azure-core:

azure-sdk-for-java/sdk/identity/azure-identity/src/main/java/com/azure/identity/implementation/IdentityClientBase.java

Lines 491 to 495 in 50af1d0

	if (options.getExecutorService() != null) {
	miBuilder.executorService(options.getExecutorService());
	} else {
	miBuilder.executorService(SharedExecutorService.getInstance());
	}

Why does the managed identity credential not provide the caching ability like the cachedToken of UsernamePasswordCredential?

Caching of MI tokens is a recent feature and is handled entirely at the MSAL layer.

• Why does the managed identity credential not provide the sync acquisition token method instead the default async method with block calling?

Good question. @g2vinay was there a reason we kept the async only API?

[billwert]

@g2vinay helped me remember that using the SharedExecutorService is still in beta - I will try the repro with the beta bits as well.

[moarychan]

If using this shared executor service, now the token acquisition thread ran in special thread pool, named with azure-sdk-global-thread-x