Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SecretCache doesn't support virtual threads in JDK 22/23 Linux #115

Open
InforBG opened this issue Sep 26, 2024 · 0 comments
Open

SecretCache doesn't support virtual threads in JDK 22/23 Linux #115

InforBG opened this issue Sep 26, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@InforBG
Copy link

InforBG commented Sep 26, 2024

Describe the bug
When starting the application service, the program will need to get many secrets from AWS SecretsManager.
To improve the performance, the program will load all secrets in a thread pool.
The issue is found when creating the ExecutorService instance with Executors.newVirtualThreadPerTaskExecutor() with JDK 22/23 Linux. When the number of virtual threads >=8, all threads hang for a long time (>30 seconds).

Tested following cases and they all work fine:

  • the number of virtual threads <8;
  • change to use Executors.newFixedThreadPool(20) with JDK 22/23 Linux;
  • All Windows JDK 21/22/23 work fine with the virtual threads.
  • Linux JDK 21 (openjdk-21.0.2_linux-x64_bin.tar.gz) works fine.

To Reproduce
Steps to reproduce:
Run the program as below:

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

import com.amazonaws.secretsmanager.caching.SecretCache;

public class AWSSecretCacheXTest {

	private static final FormattingLogger LOG = FormattingLogger.getLogger(AWSSecretCacheXTest.class);

	public static void main(String[] args) {
		final SecretCache cache = new SecretCache();

		// Note: the problem can be reproduced when the number of secrets >= 8.
		List<String> secretKeys = getSecretKeys();

		// The virtual thread has problem in JDK 22/23 Linux.
		var executor = Executors.newVirtualThreadPerTaskExecutor();
		// var executor = Executors.newFixedThreadPool(20); // this works fine in JDK 22/23 Linux
		try {
			for (String secretKey : secretKeys) {
				executor.submit(() -> {
					try {
						cache.getSecretString(secretKey);
					} catch (Throwable e) {
						LOG.error("Failed to get secret key '%s'.", e, secretKey);
					}
				});
			}
		} finally {
			executor.shutdown();
			try {
				if (!executor.awaitTermination(20, TimeUnit.SECONDS)) {
					executor.shutdownNow();
				}
			} catch (InterruptedException e) {
				executor.shutdownNow();
			}
		}
	}

	static List<String> getSecretKeys() {
		List<String> secretKeys = new ArrayList<>();
		secretKeys.add("secret01");
		secretKeys.add("secret02");
		// ...
		secretKeys.add("secret09");
		return secretKeys;
	}
}

Expected behavior
The SecretCache.getSecretString() method should work when using virtual threads and JDK 22/23 Linux.

Environment

OS: amazonlinux:2,
Java Version: JDK 22/23 Linux, (amazon-corretto-23.0.0.37.1-linux-x64.tar.gz, openjdk-23_linux-x64_bin.tar.gz and openjdk-22.0.2_linux-x64_bin.tar.gz)
AWS SDK version: latest version 2.28.6.
aws-secretsmanager-caching-java version: 2.0.0.

Additional context
N/A

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant