Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Likely race condition between model creation and deletion #2312

Open
dblock opened this issue Dec 6, 2024 · 4 comments
Open

[BUG] Likely race condition between model creation and deletion #2312

dblock opened this issue Dec 6, 2024 · 4 comments
Assignees
@vamshin
Labels
bug Something isn't working untriaged

Comments

@dblock
Copy link
Member

dblock commented Dec 6, 2024 •
edited
Loading

What is the bug?

Coming from opensearch-project/opensearch-api-specification#708 where I added a test that creates a model by ID then deletes it, quickly, without waiting for the model to finish training, and just retrying.

I ended up in a weird state where creation of a model fails because model already exists, yet deleting the model returns that it doesn't exist. There's a race condition somewhere.

[INFO] => POST /_plugins/_knn/models/model-1/_train ({}) [application/json] {
  "training_index": "movies",
  "training_field": "recommendation_vector",
  "dimension": 8,
  "max_training_vector_count": 1200,
  "search_size": 100,
  "description": "Test model",
  "mode": "on_disk",
  "compression_level": "32x",
  "spaceType": "l2"
}
[INFO] <= 400 (application/json) | {
  "error": {
    "root_cause": [
      {
        "type": "action_request_validation_exception",
        "reason": "Validation Failed: 1: Model with id=\"model-1\" already exists;"
      }
    ],
    "type": "action_request_validation_exception",
    "reason": "Validation Failed: 1: Model with id=\"model-1\" already exists;"
  },
  "status": 400
}

[INFO] => DELETE /_plugins/_knn/models/model-1 ({}) [application/json] 
[INFO] <= 404 (application/json) | {
  "error": {
    "root_cause": [
      {
        "type": "resource_not_found_exception",
        "reason": "Unable to delete model [model-1]. Model does not exist"
      }
    ],
    "type": "resource_not_found_exception",
    "reason": "Unable to delete model [model-1]. Model does not exist"
  },
  "status": 404
}

What is the expected behavior?

It should not be possible to get into this state.

What is your host/environment?

2.18 docker, Mac OSX

Do you have any additional context?

Here's the log from this docker run.

osx ~/source/opensearch-project/opensearch-api-specification/dblock-opensearch-api-specification/tests/default (knn-plugin)$ docker compose up
WARN[0000] The "OPENSEARCH_JAVA_OPTS" variable is not set. Defaulting to a blank string. 
WARN[0000] The "OPENSEARCH_DOCKER_REF" variable is not set. Defaulting to a blank string. 
WARN[0000] /Users/dblock/source/opensearch-project/opensearch-api-specification/dblock-opensearch-api-specification/tests/default/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 1/0
 ✔ Container default-opensearch-cluster-1  Created                                                                                                                                    0.0s 
Attaching to opensearch-cluster-1
opensearch-cluster-1  | Enabling OpenSearch Security Plugin
opensearch-cluster-1  | Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin 
opensearch-cluster-1  | OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user. 
opensearch-cluster-1  | Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string. 
opensearch-cluster-1  | If a password is not provided, the setup will quit. 
opensearch-cluster-1  |  For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
opensearch-cluster-1  | ### OpenSearch Security Demo Installer
opensearch-cluster-1  | ### ** Warning: Do not use on production or public reachable systems **
opensearch-cluster-1  | OpenSearch install type: rpm/deb on Linux 6.10.11-linuxkit aarch64
opensearch-cluster-1  | OpenSearch config dir: /usr/share/opensearch/config/
opensearch-cluster-1  | OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
opensearch-cluster-1  | OpenSearch bin dir: /usr/share/opensearch/bin/
opensearch-cluster-1  | OpenSearch plugins dir: /usr/share/opensearch/plugins/
opensearch-cluster-1  | OpenSearch lib dir: /usr/share/opensearch/lib/
opensearch-cluster-1  | Detected OpenSearch Version: 2.18.0
opensearch-cluster-1  | Detected OpenSearch Security Version: 2.18.0.0
opensearch-cluster-1  | /usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
opensearch-cluster-1  | Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
opensearch-cluster-1  | WARNING: Using incubator modules: jdk.incubator.vector
opensearch-cluster-1  | WARNING: A terminally deprecated method in java.lang.System has been called
opensearch-cluster-1  | WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.18.0.jar)
opensearch-cluster-1  | WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
opensearch-cluster-1  | WARNING: System::setSecurityManager will be removed in a future release
opensearch-cluster-1  | Dec 06, 2024 6:38:06 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
opensearch-cluster-1  | WARNING: COMPAT locale provider will be removed in a future release
opensearch-cluster-1  | WARNING: A terminally deprecated method in java.lang.System has been called
opensearch-cluster-1  | WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.18.0.jar)
opensearch-cluster-1  | WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
opensearch-cluster-1  | WARNING: System::setSecurityManager will be removed in a future release
opensearch-cluster-1  | [2024-12-06T18:38:06,374][INFO ][o.o.n.Node               ] [1eeceff03e18] version[2.18.0], pid[1], build[tar/99a9a81da366173b0c2b963b26ea92e15ef34547/2024-10-31T19:08:04.231254959Z], OS[Linux/6.10.11-linuxkit/aarch64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.5/21.0.5+11-LTS]
opensearch-cluster-1  | [2024-12-06T18:38:06,375][INFO ][o.o.n.Node               ] [1eeceff03e18] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
opensearch-cluster-1  | [2024-12-06T18:38:06,375][INFO ][o.o.n.Node               ] [1eeceff03e18] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-14437491619738004252, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
opensearch-cluster-1  | [2024-12-06T18:38:06,482][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [1eeceff03e18] Java vector incubator API enabled; uses preferredBitSize=128; FMA enabled
opensearch-cluster-1  | [2024-12-06T18:38:06,923][INFO ][o.o.s.s.t.SSLConfig      ] [1eeceff03e18] SSL dual mode is disabled
opensearch-cluster-1  | [2024-12-06T18:38:06,923][INFO ][o.o.s.OpenSearchSecurityPlugin] [1eeceff03e18] OpenSearch Config path is /usr/share/opensearch/config
opensearch-cluster-1  | [2024-12-06T18:38:06,986][WARN ][o.o.s.s.SslSettingsManager] [1eeceff03e18] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of {}
opensearch-cluster-1  | java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSLContext
opensearch-cluster-1  | 	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
opensearch-cluster-1  | 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593) ~[?:?]
opensearch-cluster-1  | 	at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
opensearch-cluster-1  | 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?]
opensearch-cluster-1  | 	at java.base/java.lang.Class.forName0(Native Method) ~[?:?]
opensearch-cluster-1  | 	at java.base/java.lang.Class.forName(Class.java:534) ~[?:?]
opensearch-cluster-1  | 	at java.base/java.lang.Class.forName(Class.java:513) ~[?:?]
opensearch-cluster-1  | 	at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:95) ~[netty-handler-4.1.114.Final.jar:4.1.114.Final]
opensearch-cluster-1  | 	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin$4.run(OpenSearchSecuritySSLPlugin.java:218) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
opensearch-cluster-1  | 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
opensearch-cluster-1  | 	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:213) [opensearch-security-2.18.0.0.jar:2.18.0.0]
opensearch-cluster-1  | 	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) [opensearch-security-2.18.0.0.jar:2.18.0.0]
opensearch-cluster-1  | 	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
opensearch-cluster-1  | 	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
opensearch-cluster-1  | 	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
opensearch-cluster-1  | 	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.node.Node.<init>(Node.java:523) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.node.Node.<init>(Node.java:450) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | 	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.18.0.jar:2.18.0]
opensearch-cluster-1  | [2024-12-06T18:38:07,055][INFO ][o.o.s.s.SslSettingsManager] [1eeceff03e18] TLS HTTP Provider                    : JDK
opensearch-cluster-1  | [2024-12-06T18:38:07,055][INFO ][o.o.s.s.SslSettingsManager] [1eeceff03e18] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
opensearch-cluster-1  | [2024-12-06T18:38:07,056][INFO ][o.o.s.s.SslSettingsManager] [1eeceff03e18] TLS Transport Client Provider             : JDK
opensearch-cluster-1  | [2024-12-06T18:38:07,056][INFO ][o.o.s.s.SslSettingsManager] [1eeceff03e18] TLS Transport Server Provider             : JDK
opensearch-cluster-1  | [2024-12-06T18:38:07,056][INFO ][o.o.s.s.SslSettingsManager] [1eeceff03e18] Enabled TLS protocols for Transport layer : [TLSv1.3, TLSv1.2]
opensearch-cluster-1  | [2024-12-06T18:38:07,405][INFO ][o.o.s.OpenSearchSecurityPlugin] [1eeceff03e18] Clustername: docker-cluster
opensearch-cluster-1  | [2024-12-06T18:38:07,409][WARN ][o.o.s.OpenSearchSecurityPlugin] [1eeceff03e18] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
opensearch-cluster-1  | [2024-12-06T18:38:07,410][WARN ][o.o.s.OpenSearchSecurityPlugin] [1eeceff03e18] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
opensearch-cluster-1  | [2024-12-06T18:38:07,410][WARN ][o.o.s.OpenSearchSecurityPlugin] [1eeceff03e18] File /usr/share/opensearch/config/securityadmin_demo.sh has insecure file permissions (should be 0600)
opensearch-cluster-1  | [2024-12-06T18:38:07,410][WARN ][o.o.s.OpenSearchSecurityPlugin] [1eeceff03e18] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
opensearch-cluster-1  | [2024-12-06T18:38:07,410][WARN ][o.o.s.OpenSearchSecurityPlugin] [1eeceff03e18] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600)
opensearch-cluster-1  | [2024-12-06T18:38:07,410][WARN ][o.o.s.OpenSearchSecurityPlugin] [1eeceff03e18] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
opensearch-cluster-1  | [2024-12-06T18:38:07,410][WARN ][o.o.s.OpenSearchSecurityPlugin] [1eeceff03e18] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600)
opensearch-cluster-1  | [2024-12-06T18:38:07,830][INFO ][o.o.p.c.c.PluginSettings ] [1eeceff03e18] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
opensearch-cluster-1  | [2024-12-06T18:38:08,161][INFO ][o.o.i.r.ReindexPlugin    ] [1eeceff03e18] ReindexPlugin reloadSPI called
opensearch-cluster-1  | [2024-12-06T18:38:08,161][INFO ][o.o.i.r.ReindexPlugin    ] [1eeceff03e18] Unable to find any implementation for RemoteReindexExtension
opensearch-cluster-1  | [2024-12-06T18:38:08,175][INFO ][o.o.j.JobSchedulerPlugin ] [1eeceff03e18] Loaded scheduler extension: opensearch_time_series_analytics, index: .opendistro-anomaly-detector-jobs
opensearch-cluster-1  | [2024-12-06T18:38:08,187][INFO ][o.o.j.JobSchedulerPlugin ] [1eeceff03e18] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
opensearch-cluster-1  | [2024-12-06T18:38:08,188][INFO ][o.o.j.JobSchedulerPlugin ] [1eeceff03e18] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
opensearch-cluster-1  | [2024-12-06T18:38:08,189][INFO ][o.o.j.JobSchedulerPlugin ] [1eeceff03e18] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
opensearch-cluster-1  | [2024-12-06T18:38:08,189][INFO ][o.o.j.JobSchedulerPlugin ] [1eeceff03e18] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job
opensearch-cluster-1  | [2024-12-06T18:38:08,190][INFO ][o.o.j.JobSchedulerPlugin ] [1eeceff03e18] Loaded scheduler extension: async-query-scheduler, index: .async-query-scheduler
opensearch-cluster-1  | [2024-12-06T18:38:08,203][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [aggs-matrix-stats]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [analysis-common]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [cache-common]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [geo]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [ingest-common]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [ingest-geoip]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [ingest-user-agent]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [lang-expression]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [lang-mustache]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [lang-painless]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [mapper-extras]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [opensearch-dashboards]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [parent-join]
opensearch-cluster-1  | [2024-12-06T18:38:08,204][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [percolator]
opensearch-cluster-1  | [2024-12-06T18:38:08,205][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [rank-eval]
opensearch-cluster-1  | [2024-12-06T18:38:08,205][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [reindex]
opensearch-cluster-1  | [2024-12-06T18:38:08,205][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [repository-url]
opensearch-cluster-1  | [2024-12-06T18:38:08,205][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [search-pipeline-common]
opensearch-cluster-1  | [2024-12-06T18:38:08,205][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [systemd]
opensearch-cluster-1  | [2024-12-06T18:38:08,205][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded module [transport-netty4]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-alerting]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-anomaly-detection]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-asynchronous-search]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-cross-cluster-replication]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-custom-codecs]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-flow-framework]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-geospatial]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-index-management]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-job-scheduler]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-knn]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-ml]
opensearch-cluster-1  | [2024-12-06T18:38:08,206][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-neural-search]
opensearch-cluster-1  | [2024-12-06T18:38:08,207][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-notifications]
opensearch-cluster-1  | [2024-12-06T18:38:08,207][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-notifications-core]
opensearch-cluster-1  | [2024-12-06T18:38:08,207][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-observability]
opensearch-cluster-1  | [2024-12-06T18:38:08,207][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-performance-analyzer]
opensearch-cluster-1  | [2024-12-06T18:38:08,207][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-reports-scheduler]
opensearch-cluster-1  | [2024-12-06T18:38:08,207][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-security]
opensearch-cluster-1  | [2024-12-06T18:38:08,207][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-security-analytics]
opensearch-cluster-1  | [2024-12-06T18:38:08,207][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-skills]
opensearch-cluster-1  | [2024-12-06T18:38:08,207][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-sql]
opensearch-cluster-1  | [2024-12-06T18:38:08,207][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [opensearch-system-templates]
opensearch-cluster-1  | [2024-12-06T18:38:08,207][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] loaded plugin [query-insights]
opensearch-cluster-1  | [2024-12-06T18:38:08,215][INFO ][o.o.s.OpenSearchSecurityPlugin] [1eeceff03e18] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
opensearch-cluster-1  | [2024-12-06T18:38:08,249][DEPRECATION][o.o.d.c.s.Settings       ] [1eeceff03e18] [node.max_local_storage_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
opensearch-cluster-1  | [2024-12-06T18:38:08,257][WARN ][stderr                   ] [1eeceff03e18] WARNING: A restricted method in java.lang.foreign.Linker has been called
opensearch-cluster-1  | [2024-12-06T18:38:08,257][WARN ][stderr                   ] [1eeceff03e18] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
opensearch-cluster-1  | [2024-12-06T18:38:08,257][WARN ][stderr                   ] [1eeceff03e18] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
opensearch-cluster-1  | [2024-12-06T18:38:08,313][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [1eeceff03e18] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
opensearch-cluster-1  | [2024-12-06T18:38:08,318][INFO ][o.o.e.NodeEnvironment    ] [1eeceff03e18] using [1] data paths, mounts [[/ (overlay)]], net usable_space [43gb], net total_space [58.3gb], types [overlay]
opensearch-cluster-1  | [2024-12-06T18:38:08,318][INFO ][o.o.e.NodeEnvironment    ] [1eeceff03e18] heap size [1gb], compressed ordinary object pointers [true]
opensearch-cluster-1  | [2024-12-06T18:38:08,373][INFO ][o.o.n.Node               ] [1eeceff03e18] node name [1eeceff03e18], node ID [II4924tpRYav01CECtR9hg], cluster name [docker-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
opensearch-cluster-1  | [2024-12-06T18:38:08,399][INFO ][o.o.e.ExtensionsManager  ] [1eeceff03e18] ExtensionsManager initialized
opensearch-cluster-1  | [2024-12-06T18:38:09,318][DEPRECATION][o.o.d.c.s.Settings       ] [1eeceff03e18] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
opensearch-cluster-1  | [2024-12-06T18:38:09,318][DEPRECATION][o.o.d.c.s.Settings       ] [1eeceff03e18] [plugins.security.ssl.transport.enforce_hostname_verification] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
opensearch-cluster-1  | [2024-12-06T18:38:09,668][INFO ][o.o.n.p.NeuralSearch     ] [1eeceff03e18] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
opensearch-cluster-1  | [2024-12-06T18:38:09,868][WARN ][o.o.s.c.Salt             ] [1eeceff03e18] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
opensearch-cluster-1  | [2024-12-06T18:38:09,897][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Message routing enabled: true
opensearch-cluster-1  | [2024-12-06T18:38:09,919][INFO ][o.o.s.f.SecurityFilter   ] [1eeceff03e18] <NONE> indices are made immutable.
opensearch-cluster-1  | [2024-12-06T18:38:10,099][INFO ][o.o.t.b.CircuitBreakerService] [1eeceff03e18] Registered memory breaker.
opensearch-cluster-1  | [2024-12-06T18:38:10,345][INFO ][o.o.m.b.MLCircuitBreakerService] [1eeceff03e18] Registered ML memory breaker.
opensearch-cluster-1  | [2024-12-06T18:38:10,345][INFO ][o.o.m.b.MLCircuitBreakerService] [1eeceff03e18] Registered ML disk breaker.
opensearch-cluster-1  | [2024-12-06T18:38:10,345][INFO ][o.o.m.b.MLCircuitBreakerService] [1eeceff03e18] Registered ML native memory breaker.
opensearch-cluster-1  | [2024-12-06T18:38:10,404][INFO ][o.r.Reflections          ] [1eeceff03e18] Reflections took 31 ms to scan 1 urls, producing 26 keys and 67 values 
opensearch-cluster-1  | [2024-12-06T18:38:10,419][INFO ][o.r.Reflections          ] [1eeceff03e18] Reflections took 1 ms to scan 1 urls, producing 3 keys and 5 values 
opensearch-cluster-1  | [2024-12-06T18:38:10,516][WARN ][o.o.s.p.SQLPlugin        ] [1eeceff03e18] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
opensearch-cluster-1  | [2024-12-06T18:38:10,976][INFO ][o.o.t.NettyAllocator     ] [1eeceff03e18] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
opensearch-cluster-1  | [2024-12-06T18:38:10,979][INFO ][o.o.s.s.t.SSLConfig      ] [1eeceff03e18] SSL dual mode is disabled
opensearch-cluster-1  | [2024-12-06T18:38:11,047][INFO ][o.o.d.DiscoveryModule    ] [1eeceff03e18] using discovery type [single-node] and seed hosts providers [settings]
opensearch-cluster-1  | [2024-12-06T18:38:11,377][WARN ][o.o.g.DanglingIndicesState] [1eeceff03e18] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
opensearch-cluster-1  | [2024-12-06T18:38:11,766][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [1eeceff03e18] PerformanceAnalyzer Enabled: false
opensearch-cluster-1  | [2024-12-06T18:38:11,794][INFO ][o.o.n.Node               ] [1eeceff03e18] initialized
opensearch-cluster-1  | [2024-12-06T18:38:11,794][INFO ][o.o.n.Node               ] [1eeceff03e18] starting ...
opensearch-cluster-1  | [2024-12-06T18:38:11,815][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [windows_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,816][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [waf_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,816][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [vpcflow_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,816][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [test_windows_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,816][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [s3_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,817][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [others_web_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,817][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [others_proxy_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,817][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [others_macos_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,817][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [others_compliance_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,818][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [others_cloud_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,818][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [others_apt_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,818][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [others_application_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,818][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [okta_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,818][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [network_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,819][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [netflow_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,819][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [m365_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,819][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [linux_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,819][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [gworkspace_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,820][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [github_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,820][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [dns_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,821][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [cloudtrail_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,821][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [azure_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,821][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [apache_access_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,821][INFO ][o.o.s.l.BuiltinLogTypeLoader] [1eeceff03e18] Loaded [ad_ldap_logtype.json] log type
opensearch-cluster-1  | [2024-12-06T18:38:11,895][INFO ][o.o.t.TransportService   ] [1eeceff03e18] publish_address {172.20.0.2:9300}, bound_addresses {[::]:9300}
opensearch-cluster-1  | [2024-12-06T18:38:11,896][INFO ][o.o.t.TransportService   ] [1eeceff03e18] Remote clusters initialized successfully.
opensearch-cluster-1  | [2024-12-06T18:38:12,019][INFO ][o.o.c.c.Coordinator      ] [1eeceff03e18] cluster UUID [t3SKPp7ERa2JV5k080d77g]
opensearch-cluster-1  | [2024-12-06T18:38:12,134][INFO ][o.o.c.s.MasterService    ] [1eeceff03e18] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper and count: 3, term: 2, version: 137, delta: cluster-manager node changed {previous [], current [{1eeceff03e18}{II4924tpRYav01CECtR9hg}{O6L6c3m6RKmezaFrAcEdvA}{172.20.0.2}{172.20.0.2:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
opensearch-cluster-1  | [2024-12-06T18:38:12,164][INFO ][o.o.c.s.ClusterApplierService] [1eeceff03e18] cluster-manager node changed {previous [], current [{1eeceff03e18}{II4924tpRYav01CECtR9hg}{O6L6c3m6RKmezaFrAcEdvA}{172.20.0.2}{172.20.0.2:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 2, version: 137, reason: Publication{term=2, version=137}
opensearch-cluster-1  | [2024-12-06T18:38:12,169][INFO ][o.o.t.i.IndexManagement  ] [1eeceff03e18] Candidate custom result indices are empty.
opensearch-cluster-1  | [2024-12-06T18:38:12,170][INFO ][o.o.t.i.IndexManagement  ] [1eeceff03e18] Candidate custom result indices are empty.
opensearch-cluster-1  | [2024-12-06T18:38:12,170][INFO ][o.o.t.c.ClusterEventListener] [1eeceff03e18] Cluster is not recovered yet.
opensearch-cluster-1  | [2024-12-06T18:38:12,185][INFO ][o.o.i.i.ManagedIndexCoordinator] [1eeceff03e18] Cache cluster manager node onClusterManager time: 1733510292185
opensearch-cluster-1  | [2024-12-06T18:38:12,190][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [1eeceff03e18] Config override setting update called with empty string. Ignoring.
opensearch-cluster-1  | [2024-12-06T18:38:12,192][DEPRECATION][o.o.d.c.u.c.ThreadContext] [1eeceff03e18] Default access to markAsSystemContext will be removed in a future release. Permission to use markAsSystemContext must be explicitly granted.
opensearch-cluster-1  | [2024-12-06T18:38:12,195][INFO ][o.o.d.PeerFinder         ] [1eeceff03e18] setting findPeersInterval to [1s] as node commission status = [true] for local node [{1eeceff03e18}{II4924tpRYav01CECtR9hg}{O6L6c3m6RKmezaFrAcEdvA}{172.20.0.2}{172.20.0.2:9300}{dimr}{shard_indexing_pressure_enabled=true}]
opensearch-cluster-1  | [2024-12-06T18:38:12,197][WARN ][o.o.c.r.a.AllocationService] [1eeceff03e18] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
opensearch-cluster-1  | [2024-12-06T18:38:12,198][INFO ][o.o.h.AbstractHttpServerTransport] [1eeceff03e18] publish_address {172.20.0.2:9200}, bound_addresses {[::]:9200}
opensearch-cluster-1  | [2024-12-06T18:38:12,198][INFO ][o.o.n.Node               ] [1eeceff03e18] started
opensearch-cluster-1  | [2024-12-06T18:38:12,198][INFO ][o.o.s.c.ConfigurationRepository] [1eeceff03e18] Will attempt to create index .opendistro_security and default configs if they are absent
opensearch-cluster-1  | [2024-12-06T18:38:12,198][INFO ][o.o.s.c.ConfigurationRepository] [1eeceff03e18] Background init thread started. Install default config?: true
opensearch-cluster-1  | [2024-12-06T18:38:12,199][INFO ][o.o.s.c.ConfigurationRepository] [1eeceff03e18] Wait for cluster to be available ...
opensearch-cluster-1  | [2024-12-06T18:38:12,199][INFO ][o.o.s.OpenSearchSecurityPlugin] [1eeceff03e18] 0 OpenSearch Security modules loaded so far: []
opensearch-cluster-1  | [2024-12-06T18:38:12,204][WARN ][o.o.c.r.a.AllocationService] [1eeceff03e18] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
opensearch-cluster-1  | [2024-12-06T18:38:12,229][INFO ][o.o.c.s.ClusterSettings  ] [1eeceff03e18] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
opensearch-cluster-1  | [2024-12-06T18:38:12,233][INFO ][o.o.t.c.HashRing         ] [1eeceff03e18] Node added: [II4924tpRYav01CECtR9hg]
opensearch-cluster-1  | [2024-12-06T18:38:12,236][INFO ][o.o.t.c.HashRing         ] [1eeceff03e18] Add data node to version hash ring: II4924tpRYav01CECtR9hg
opensearch-cluster-1  | [2024-12-06T18:38:12,237][INFO ][o.o.t.c.HashRing         ] [1eeceff03e18] All nodes with known version: {II4924tpRYav01CECtR9hg=ADNodeInfo{version=2.18.0, isEligibleDataNode=true}}
opensearch-cluster-1  | [2024-12-06T18:38:12,237][INFO ][o.o.t.c.HashRing         ] [1eeceff03e18] Rebuild hash ring for realtime with cooldown, nodeChangeEvents size 0
opensearch-cluster-1  | [2024-12-06T18:38:12,237][INFO ][o.o.t.c.HashRing         ] [1eeceff03e18] Build version hash ring successfully
opensearch-cluster-1  | [2024-12-06T18:38:12,238][INFO ][o.o.t.c.ADDataMigrator   ] [1eeceff03e18] Start migrating AD data
opensearch-cluster-1  | [2024-12-06T18:38:12,238][INFO ][o.o.t.c.ADDataMigrator   ] [1eeceff03e18] AD job index doesn't exist, no need to migrate
opensearch-cluster-1  | [2024-12-06T18:38:12,238][INFO ][o.o.t.c.ClusterEventListener] [1eeceff03e18] Init version hash ring successfully
opensearch-cluster-1  | [2024-12-06T18:38:12,248][INFO ][o.o.g.GatewayService     ] [1eeceff03e18] recovered [6] indices into cluster_state
opensearch-cluster-1  | [2024-12-06T18:38:12,249][WARN ][o.o.o.i.ObservabilityIndex] [1eeceff03e18] message: index [.opensearch-observability/Zjx9GpDfTdu43DSC7Zgp7A] already exists
opensearch-cluster-1  | [2024-12-06T18:38:12,250][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Loading builtin types!
opensearch-cluster-1  | [2024-12-06T18:38:12,251][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Indexing [429] fieldMappingDocs from logTypes: 24
opensearch-cluster-1  | [2024-12-06T18:38:12,268][WARN ][o.o.c.r.a.AllocationService] [1eeceff03e18] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
opensearch-cluster-1  | [2024-12-06T18:38:12,272][WARN ][o.o.s.SecurityAnalyticsPlugin] [1eeceff03e18] Failed to initialize LogType config index and builtin log types
opensearch-cluster-1  | [2024-12-06T18:38:12,328][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] PluginService:onIndexModule index:[.opendistro_security/ykseYRELThuhDzkHs8xDLQ]
opensearch-cluster-1  | [2024-12-06T18:38:12,402][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] PluginService:onIndexModule index:[.opensearch-knn-models/_XxheXscRSOupdQb0Nh-Tw]
opensearch-cluster-1  | [2024-12-06T18:38:12,413][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] PluginService:onIndexModule index:[.plugins-ml-config/t0MLVpVVSamBHuviG0L1Dg]
opensearch-cluster-1  | [2024-12-06T18:38:12,419][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] PluginService:onIndexModule index:[.opensearch-observability/Zjx9GpDfTdu43DSC7Zgp7A]
opensearch-cluster-1  | [2024-12-06T18:38:12,551][WARN ][o.o.c.r.a.AllocationService] [1eeceff03e18] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
opensearch-cluster-1  | [2024-12-06T18:38:12,571][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] PluginService:onIndexModule index:[security-auditlog-2024.12.06/WvanWXguQemuS1fFt_KZdg]
opensearch-cluster-1  | [2024-12-06T18:38:12,578][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/4IY0jXKWSMml-RZO7_olEQ]
opensearch-cluster-1  | [2024-12-06T18:38:12,623][INFO ][o.o.c.r.a.AllocationService] [1eeceff03e18] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.opensearch-sap-log-types-config][0]]]).
opensearch-cluster-1  | [2024-12-06T18:38:12,640][WARN ][o.o.c.r.a.AllocationService] [1eeceff03e18] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
opensearch-cluster-1  | [2024-12-06T18:38:13,196][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Loading builtin types!
opensearch-cluster-1  | [2024-12-06T18:38:13,196][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Indexing [429] fieldMappingDocs from logTypes: 24
opensearch-cluster-1  | [2024-12-06T18:38:13,198][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Loading builtin types!
opensearch-cluster-1  | [2024-12-06T18:38:13,198][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Indexing [429] fieldMappingDocs from logTypes: 24
opensearch-cluster-1  | [2024-12-06T18:38:13,200][INFO ][o.o.s.i.DetectorIndexManagementService] [1eeceff03e18] info deleteOldIndices
opensearch-cluster-1  | [2024-12-06T18:38:13,201][INFO ][o.o.s.c.ConfigurationRepository] [1eeceff03e18] Index .opendistro_security already exists
opensearch-cluster-1  | [2024-12-06T18:38:13,201][INFO ][o.o.s.c.ConfigurationRepository] [1eeceff03e18] Node started, try to initialize it. Wait for at least yellow cluster state....
opensearch-cluster-1  | [2024-12-06T18:38:13,202][INFO ][o.o.s.i.DetectorIndexManagementService] [1eeceff03e18] No Old Correlation Indices to delete
opensearch-cluster-1  | [2024-12-06T18:38:13,203][INFO ][o.o.s.i.DetectorIndexManagementService] [1eeceff03e18] info deleteOldIndices
opensearch-cluster-1  | [2024-12-06T18:38:13,203][INFO ][o.o.s.i.DetectorIndexManagementService] [1eeceff03e18] No Old IOC Findings Indices to delete
opensearch-cluster-1  | [2024-12-06T18:38:13,204][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Will update 'config' with /usr/share/opensearch/config/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-cluster-1  | [2024-12-06T18:38:13,255][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Index .opendistro_security already contains doc with id config, skipping update.
opensearch-cluster-1  | [2024-12-06T18:38:13,256][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Will update 'roles' with /usr/share/opensearch/config/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-cluster-1  | [2024-12-06T18:38:13,263][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Index .opendistro_security already contains doc with id roles, skipping update.
opensearch-cluster-1  | [2024-12-06T18:38:13,263][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Will update 'rolesmapping' with /usr/share/opensearch/config/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-cluster-1  | [2024-12-06T18:38:13,269][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Index .opendistro_security already contains doc with id rolesmapping, skipping update.
opensearch-cluster-1  | [2024-12-06T18:38:13,269][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Will update 'internalusers' with /usr/share/opensearch/config/opensearch-security/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-cluster-1  | [2024-12-06T18:38:13,276][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Index .opendistro_security already contains doc with id internalusers, skipping update.
opensearch-cluster-1  | [2024-12-06T18:38:13,276][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Will update 'actiongroups' with /usr/share/opensearch/config/opensearch-security/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-cluster-1  | [2024-12-06T18:38:13,280][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Index .opendistro_security already contains doc with id actiongroups, skipping update.
opensearch-cluster-1  | [2024-12-06T18:38:13,280][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Will update 'tenants' with /usr/share/opensearch/config/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-cluster-1  | [2024-12-06T18:38:13,283][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Index .opendistro_security already contains doc with id tenants, skipping update.
opensearch-cluster-1  | [2024-12-06T18:38:13,283][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Will update 'nodesdn' with /usr/share/opensearch/config/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
opensearch-cluster-1  | [2024-12-06T18:38:13,285][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Index .opendistro_security already contains doc with id nodesdn, skipping update.
opensearch-cluster-1  | [2024-12-06T18:38:13,285][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Will update 'whitelist' with /usr/share/opensearch/config/opensearch-security/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
opensearch-cluster-1  | [2024-12-06T18:38:13,286][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Index .opendistro_security already contains doc with id whitelist, skipping update.
opensearch-cluster-1  | [2024-12-06T18:38:13,286][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Will update 'allowlist' with /usr/share/opensearch/config/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
opensearch-cluster-1  | [2024-12-06T18:38:13,288][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Index .opendistro_security already contains doc with id allowlist, skipping update.
opensearch-cluster-1  | [2024-12-06T18:38:13,288][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Will update 'audit' with /usr/share/opensearch/config/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
opensearch-cluster-1  | [2024-12-06T18:38:13,297][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Indexing [428] fieldMappingDocs
opensearch-cluster-1  | [2024-12-06T18:38:13,297][INFO ][o.o.s.s.ConfigHelper     ] [1eeceff03e18] Index .opendistro_security already contains doc with id audit, skipping update.
opensearch-cluster-1  | [2024-12-06T18:38:13,299][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Indexing [428] fieldMappingDocs
opensearch-cluster-1  | [2024-12-06T18:38:13,396][INFO ][stdout                   ] [1eeceff03e18] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][stdout                   ] [1eeceff03e18] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing on REST API is enabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing on Transport API is enabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing of request body is enabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Bulk requests resolution is disabled during request auditing.
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Index resolution is enabled during request auditing.
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Sensitive headers exclusion from auditing is enabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing requests from kibanaserver users is disabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing request headers <NONE> is disabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,397][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing request url params <NONE> is disabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,400][WARN ][o.o.s.a.r.AuditMessageRouter] [1eeceff03e18] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing of external configuration is disabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing of internal configuration is enabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing only metadata information for read request is enabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing will watch {} for read requests.
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing read operation requests from kibanaserver users is disabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing only metadata information for write request is enabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing diffs for write requests is disabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing write operation requests from kibanaserver users is disabled.
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Auditing will watch <NONE> for write requests.
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] .opendistro_security is used as internal security index.
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.a.i.AuditLogImpl   ] [1eeceff03e18] Internal index used for posting audit logs is null
opensearch-cluster-1  | [2024-12-06T18:38:13,400][INFO ][o.o.s.c.ConfigurationRepository] [1eeceff03e18] Hot-reloading of audit configuration is enabled
opensearch-cluster-1  | [2024-12-06T18:38:13,401][INFO ][o.o.s.c.ConfigurationRepository] [1eeceff03e18] Node '1eeceff03e18' initialized
opensearch-cluster-1  | [2024-12-06T18:38:13,601][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Loaded [428] field mapping docs successfully!
opensearch-cluster-1  | [2024-12-06T18:38:13,605][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Loaded [428] field mapping docs successfully!
opensearch-cluster-1  | [2024-12-06T18:38:13,621][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Indexing [23] customLogTypes
opensearch-cluster-1  | [2024-12-06T18:38:13,627][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Indexing [23] customLogTypes
opensearch-cluster-1  | [2024-12-06T18:38:13,645][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Loaded [23] customLogType docs successfully!
opensearch-cluster-1  | [2024-12-06T18:38:13,653][INFO ][o.o.s.l.LogTypeService   ] [1eeceff03e18] Loaded [23] customLogType docs successfully!
opensearch-cluster-1  | [2024-12-06T18:38:13,692][INFO ][o.o.s.i.DetectorIndexManagementService] [1eeceff03e18] info deleteOldIndices
opensearch-cluster-1  | [2024-12-06T18:38:13,692][INFO ][o.o.s.i.DetectorIndexManagementService] [1eeceff03e18] info deleteOldIndices
opensearch-cluster-1  | [2024-12-06T18:38:13,693][INFO ][o.o.s.i.DetectorIndexManagementService] [1eeceff03e18] No Old Finding Indices to delete
opensearch-cluster-1  | [2024-12-06T18:38:13,693][INFO ][o.o.s.i.DetectorIndexManagementService] [1eeceff03e18] No Old Alert Indices to delete
opensearch-cluster-1  | [2024-12-06T18:38:21,249][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] PluginService:onIndexModule index:[movies/biotIAG5S8-zYwWn3gbkuA]
opensearch-cluster-1  | [2024-12-06T18:38:21,262][INFO ][o.o.c.m.MetadataCreateIndexService] [1eeceff03e18] [movies] creating index, cause [api], templates [], shards [1]/[1]
opensearch-cluster-1  | [2024-12-06T18:38:21,263][WARN ][o.o.c.r.a.AllocationService] [1eeceff03e18] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
opensearch-cluster-1  | [2024-12-06T18:38:21,277][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] PluginService:onIndexModule index:[movies/biotIAG5S8-zYwWn3gbkuA]
opensearch-cluster-1  | [2024-12-06T18:38:21,316][WARN ][o.o.c.r.a.AllocationService] [1eeceff03e18] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
opensearch-cluster-1  | [2024-12-06T18:38:21,346][INFO ][o.o.p.PluginsService     ] [1eeceff03e18] PluginService:onIndexModule index:[movies/biotIAG5S8-zYwWn3gbkuA]
opensearch-cluster-1  | [2024-12-06T18:38:21,350][INFO ][o.o.c.m.MetadataMappingService] [1eeceff03e18] [movies/biotIAG5S8-zYwWn3gbkuA] update_mapping [_doc]
opensearch-cluster-1  | [2024-12-06T18:38:21,373][INFO ][o.o.k.i.c.K.NativeEngines990KnnVectorsWriter] [1eeceff03e18] Skip building vector data structure for field: recommendation_vector, as liveDoc: 2 is less than the threshold 15000 during flush
opensearch-cluster-1  | [2024-12-06T18:38:21,429][INFO ][o.o.c.m.MetadataDeleteIndexService] [1eeceff03e18] [movies/biotIAG5S8-zYwWn3gbkuA] deleting index
opensearch-cluster-1  | [2024-12-06T18:38:21,431][WARN ][o.o.c.r.a.AllocationService] [1eeceff03e18] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
opensearch-cluster-1  | [2024-12-06T18:38:21,480][ERROR][o.o.k.i.ModelDao         ] [1eeceff03e18] Failed to remove " model-1 " from Model Graveyard
opensearch-cluster-1  | Unable to delete model [model-1]. Model does not exist
opensearch-cluster-1  | [2024-12-06T18:38:21,481][ERROR][o.o.k.p.t.DeleteModelTransportAction] [1eeceff03e18] ResourceNotFoundException[Unable to delete model [model-1]. Model does not exist]
opensearch-cluster-1  | [2024-12-06T18:38:22,194][INFO ][o.o.m.a.MLModelAutoReDeployer] [1eeceff03e18] Index not found, not performing auto reloading!
opensearch-cluster-1  | [2024-12-06T18:38:22,194][INFO ][o.o.m.c.MLCommonsClusterManagerEventListener] [1eeceff03e18] Starting ML sync up job...
@dblock dblock added bug Something isn't working untriaged labels Dec 6, 2024
@navneet1v navneet1v moved this to 2.19.0 in Vector Search RoadMap Dec 10, 2024
@navneet1v
Copy link
Collaborator

@dblock thanks for creating the issue. I think from k-NN plugin we should throw exception in cases for deletion when model is in training. I remember 2 year or back there were some work happening around this. @naveentatikonda if I am not wrong you doing something like that. I might be completely wrong here. Can you please check this once if this relates to the work you were doing earlier?

@naveentatikonda
Copy link
Member

@dblock @navneet1v I guess what's happening with this test case is as soon as the training request has been submitted you triggered a delete model request so that model details has not yet been stored into the model system index. When we run the delete model request, it first tries to fetch the model information from the model system index using the GET model API but it is failing to find the model info here and throwing "Unable to delete model [model-1]. Model does not exist".

If we trigger the delete model request after some time(few sec) where the model info is stored in the system index and entered into TRAINING state, then it will be able to fetch the model info and reject the request with "Cannot delete model [model-1]. Model is still in training" exception.

If the metadata already exists for a modelID then it is expected to reject a new train model request with the same model id.

PR - #424

@navneet1v
Copy link
Collaborator

@naveentatikonda thanks for sharing the details. I do think yes it can be possible that model information is not stored in the index before a delete call comes in. This begs a question, should we make this GET model call more consistent so that it can handle these cases too. Or do you think thats not needed for now. @dblock what your thought on this?

If we trigger the delete model request after some time(few sec) where the model info is stored in the system index and entered into TRAINING state

This was added to ensure that Tests are successful.

@heemin32
Copy link
Collaborator

Based on the issue description, the reported errors are either "reason": "Validation Failed: 1: Model with id=\"model-1\" already exists;" or "Unable to delete model [model-1]. Model does not exist." There is no mention of "Cannot delete model [model-1]. Model is still in training."

This implies that even if the user waits for a long time, they are unable to create or delete model-1.

One possible explanation is that the model data is inconsistently stored—either in the cluster state or the index, but not in both—leading to failures during both creation and deletion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vamshin vamshin

Labels
bug Something isn't working untriaged
Projects
Vector Search RoadMap
Status: 2.19.0
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dblock @heemin32 @navneet1v @vamshin @naveentatikonda