Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] 2.18 requires both configs for PKCS setup plugins.security.ssl.transport.keystore_password and ...keystore_keypassword #4961

Open
AntonEliatra opened this issue Dec 9, 2024 · 4 comments
Open

[BUG] 2.18 requires both configs for PKCS setup plugins.security.ssl.transport.keystore_password and ...keystore_keypassword #4961

AntonEliatra opened this issue Dec 9, 2024 · 4 comments
Assignees
@willyborankin
Labels
bug Something isn't working triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable.

Comments

@AntonEliatra
Copy link

What is the bug?
OS 2.18 requires both configs for PKCS setup plugins.security.ssl.transport.keystore_password and ...keystore_keypassword

How can one reproduce the bug?
Remove one of the 2 lines (keystore_password or keystore_keypassword) and try to start the cluster:

`plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.keystore_filepath: opensearch-node.jks
plugins.security.ssl.transport.keystore_alias: opensearch-node
plugins.security.ssl.transport.keystore_password:
plugins.security.ssl.transport.keystore_keypassword: #This is required option in 2.18.0
plugins.security.ssl.transport.truststore_filepath: opensearch-truststore.jks
plugins.security.ssl.transport.truststore_alias: root-ca
plugins.security.ssl.transport.truststore_password: truststorepassword

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.keystore_filepath: opensearch-node.jks
plugins.security.ssl.http.keystore_alias: opensearch-node
plugins.security.ssl.http.keystore_password:
plugins.security.ssl.http.keystore_keypassword: #This is required option in 2.18.0
plugins.security.ssl.http.truststore_filepath: opensearch-truststore.jks
plugins.security.ssl.http.truststore_alias: root-ca
plugins.security.ssl.http.truststore_password: truststorepassword`

What is the expected behavior?
Only one of these configuration is required

What is your host/environment?

  • Version 2.18.0
@AntonEliatra AntonEliatra added bug Something isn't working untriaged Require the attention of the repository maintainers and may need to be prioritized labels Dec 9, 2024
@pawelw1
Copy link

pawelw1 commented Dec 9, 2024

Error when plugins.security.ssl.transport.keystore_password option is missing

opensearch-node2_2.18.0       | [2024-12-09T10:38:54,632][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-node2] uncaught exception in thread [main]
opensearch-node2_2.18.0       | org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       | Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         ... 6 more
opensearch-node2_2.18.0       | Caused by: java.lang.reflect.InvocationTargetException
opensearch-node2_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         ... 6 more
opensearch-node2_2.18.0       | Caused by: org.opensearch.OpenSearchException: Failed to load key store from /usr/share/opensearch/config/opensearch-node.jks
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.newKeyStore(KeyStoreUtils.java:202) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration$JdkKeyStoreConfiguration.createKeyStore(KeyStoreConfiguration.java:127) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:37) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:96) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:95) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         ... 6 more
opensearch-node2_2.18.0       | Caused by: org.opensearch.OpenSearchException: Failed to load keystore from /usr/share/opensearch/config/opensearch-node.jks
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.loadKeyStore(KeyStoreUtils.java:175) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.newKeyStore(KeyStoreUtils.java:187) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration$JdkKeyStoreConfiguration.createKeyStore(KeyStoreConfiguration.java:127) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:37) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:96) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:95) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         ... 6 more
opensearch-node2_2.18.0       | Caused by: java.lang.RuntimeException: java.io.IOException: keystore password was incorrect
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.loadKeyStore(KeyStoreUtils.java:172) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.newKeyStore(KeyStoreUtils.java:187) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration$JdkKeyStoreConfiguration.createKeyStore(KeyStoreConfiguration.java:127) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:37) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:96) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:95) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         ... 6 more
opensearch-node2_2.18.0       | Caused by: java.io.IOException: keystore password was incorrect
opensearch-node2_2.18.0       |         at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2097) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:228) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.security.KeyStore.load(KeyStore.java:1500) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.loadKeyStore(KeyStoreUtils.java:169) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.newKeyStore(KeyStoreUtils.java:187) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration$JdkKeyStoreConfiguration.createKeyStore(KeyStoreConfiguration.java:127) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:37) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:96) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:95) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         ... 6 more
opensearch-node2_2.18.0       | Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
opensearch-node2_2.18.0       |         at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2097) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:228) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.security.KeyStore.load(KeyStore.java:1500) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.loadKeyStore(KeyStoreUtils.java:169) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.newKeyStore(KeyStoreUtils.java:187) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration$JdkKeyStoreConfiguration.createKeyStore(KeyStoreConfiguration.java:127) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:37) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:96) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:95) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node2_2.18.0       |         ... 6 more
opensearch-node2_2.18.0       | uncaught exception in thread [main]
opensearch-node2_2.18.0       | java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
opensearch-node2_2.18.0       | Likely root cause: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
opensearch-node2_2.18.0       |         at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2097)
opensearch-node2_2.18.0       |         at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:228)
opensearch-node2_2.18.0       |         at java.base/java.security.KeyStore.load(KeyStore.java:1500)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.loadKeyStore(KeyStoreUtils.java:169)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.newKeyStore(KeyStoreUtils.java:187)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration$JdkKeyStoreConfiguration.createKeyStore(KeyStoreConfiguration.java:127)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:37)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75)
opensearch-node2_2.18.0       |         at java.base/java.security.AccessController.doPrivileged(AccessController.java:571)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:96)
opensearch-node2_2.18.0       |         at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:95)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80)
opensearch-node2_2.18.0       |         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249)
opensearch-node2_2.18.0       |         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318)
opensearch-node2_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
opensearch-node2_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796)
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744)
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545)
opensearch-node2_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197)
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523)
opensearch-node2_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450)
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
opensearch-node2_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
opensearch-node2_2.18.0       |         <<<truncated>>>
opensearch-node2_2.18.0       | For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log

Error when plugins.security.ssl.transport.keystore_keypassword option is missing

opensearch-node1_2.18.0       | [2024-12-09T10:37:31,726][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-node1] uncaught exception in thread [main]
opensearch-node1_2.18.0       | org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       | Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         ... 6 more
opensearch-node1_2.18.0       | Caused by: java.lang.reflect.InvocationTargetException
opensearch-node1_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         ... 6 more
opensearch-node1_2.18.0       | Caused by: org.opensearch.OpenSearchException: Failed to load key store from /usr/share/opensearch/config/opensearch-node.jks
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.newKeyStore(KeyStoreUtils.java:202) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration$JdkKeyStoreConfiguration.createKeyStore(KeyStoreConfiguration.java:127) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:37) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:96) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:95) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         ... 6 more
opensearch-node1_2.18.0       | Caused by: java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
opensearch-node1_2.18.0       |         at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:454) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/sun.security.util.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:93) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.security.KeyStore.getKey(KeyStore.java:1075) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.newKeyStore(KeyStoreUtils.java:196) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration$JdkKeyStoreConfiguration.createKeyStore(KeyStoreConfiguration.java:127) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:37) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:96) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:95) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         ... 6 more
opensearch-node1_2.18.0       | Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
opensearch-node1_2.18.0       |         at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:861) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:941) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:734) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/com.sun.crypto.provider.PBES2Core.engineDoFinal(PBES2Core.java:203) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2244) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/sun.security.pkcs12.PKCS12KeyStore.lambda$engineGetKey$0(PKCS12KeyStore.java:370) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run(PKCS12KeyStore.java:257) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:361) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/sun.security.util.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:93) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.security.KeyStore.getKey(KeyStore.java:1075) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.newKeyStore(KeyStoreUtils.java:196) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration$JdkKeyStoreConfiguration.createKeyStore(KeyStoreConfiguration.java:127) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:37) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:96) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:95) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
opensearch-node1_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
opensearch-node1_2.18.0       |         ... 6 more
opensearch-node1_2.18.0       | uncaught exception in thread [main]
opensearch-node1_2.18.0       | java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
opensearch-node1_2.18.0       | Likely root cause: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
opensearch-node1_2.18.0       |         at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:861)
opensearch-node1_2.18.0       |         at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:941)
opensearch-node1_2.18.0       |         at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:734)
opensearch-node1_2.18.0       |         at java.base/com.sun.crypto.provider.PBES2Core.engineDoFinal(PBES2Core.java:203)
opensearch-node1_2.18.0       |         at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2244)
opensearch-node1_2.18.0       |         at java.base/sun.security.pkcs12.PKCS12KeyStore.lambda$engineGetKey$0(PKCS12KeyStore.java:370)
opensearch-node1_2.18.0       |         at java.base/sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run(PKCS12KeyStore.java:257)
opensearch-node1_2.18.0       |         at java.base/sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:361)
opensearch-node1_2.18.0       |         at java.base/sun.security.util.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:93)
opensearch-node1_2.18.0       |         at java.base/java.security.KeyStore.getKey(KeyStore.java:1075)
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreUtils.newKeyStore(KeyStoreUtils.java:196)
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration$JdkKeyStoreConfiguration.createKeyStore(KeyStoreConfiguration.java:127)
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.config.KeyStoreConfiguration.createKeyManagerFactory(KeyStoreConfiguration.java:37)
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.lambda$buildServerSslContext$0(SslConfiguration.java:75)
opensearch-node1_2.18.0       |         at java.base/java.security.AccessController.doPrivileged(AccessController.java:571)
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslConfiguration.buildServerSslContext(SslConfiguration.java:73)
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:42)
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslContextHandler.<init>(SslContextHandler.java:38)
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.lambda$buildSslContexts$0(SslSettingsManager.java:96)
opensearch-node1_2.18.0       |         at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:95)
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80)
opensearch-node1_2.18.0       |         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249)
opensearch-node1_2.18.0       |         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318)
opensearch-node1_2.18.0       |         at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
opensearch-node1_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
opensearch-node1_2.18.0       |         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796)
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744)
opensearch-node1_2.18.0       |         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545)
opensearch-node1_2.18.0       |         <<<truncated>>>
opensearch-node1_2.18.0       | For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.lo

@AntonEliatra AntonEliatra mentioned this issue Dec 9, 2024
Open
4 tasks
@cwperks cwperks added triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable. and removed untriaged Require the attention of the repository maintainers and may need to be prioritized labels Dec 9, 2024
@cwperks
Copy link
Member

cwperks commented Dec 9, 2024

[Triage] @willyborankin Can you take a look at this issue?

@cwperks
Copy link
Member

cwperks commented Dec 20, 2024 •
edited
Loading

@willyborankin I haven't dug into this in great depth, but is this because of the default value here? https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/ssl/config/SslCertificatesLoader.java#L77

Edit: nvm it appears to be the same in 2.17

Edit 2: Ok, I see why this is the case now. Prior to 2.18 the default for the keypassword, would take the value from the keystore password setting.

Code pointers:

  1. https://github.com/opensearch-project/security/blob/2.17/src/main/java/org/opensearch/security/ssl/DefaultSecurityKeyStore.java#L333-L336
  2. https://github.com/opensearch-project/security/blob/2.17/src/main/java/org/opensearch/security/ssl/DefaultSecurityKeyStore.java#L407

By setting plugins.security.ssl.http.keystore_password to empty, it would make the default for plugins.security.ssl.http.keystore_keypassword empty as well.

During the refactor, plugins.security.ssl.http.keystore_keypassword has an explicit default of changeit.

@willyborankin wdyt? I actually think the previous behavior makes sense because the keystore password and key password are the same by default unless otherwise specified.

@willyborankin
Copy link
Collaborator

@cwperks, you’re right—this is the root cause. Refactoring unfortunately breaks backward compatibility.

willyborankin added a commit to willyborankin/security that referenced this issue Dec 30, 2024
@willyborankin
Fix SSL config for JDK PKCS setup
7b448e5 
Fix issue opensearch-project#4961 by defaulting
the keystore_keypassword setting to the same value as the keystore_password

Signed-off-by: Andrey Pleskach <[email protected]>
@willyborankin willyborankin mentioned this issue Dec 30, 2024
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@willyborankin willyborankin

Labels
bug Something isn't working triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@willyborankin @cwperks @AntonEliatra @pawelw1