[Concurrent Segment Search] Isolation between search request for system index, search throttled index and regular index in concurrent search enabled setup

[sohami]

Is your feature request related to a problem? Please describe

In OpenSearch, we have 3 separate threadpools for search traffic "search", "search_throttled", "system_read". These threadpools handles the search requests for different indices. For example: "system_read" is used for the searches on the system index, "search" is used for searches on the regular indices where most of the user workload will be served and "search_throttled" is a mechanism to throttle searches on a specific index and run it at very low TPS. With concurrent segment search, we introduced a new threadpool index_searcher which is used to actually simulate the search tasks in lucene concurrently whereas the search thread waits for all the tasks to complete. However, with this the index_searcher threadpool will be shared for search requests between all the index types i.e. system index, regular index and search throttled index. This is removing the threadpool isolation which was present earlier.

Describe the solution you'd like

With concurrent search enabled clusters, we have 2 options:

1. [Preferred]: We can keep these isolation and use the index_searcher threadpool to serve requests for indices which are neither marked for search throttled nor system index. We want most of the performance benefits for the regular indices where majority of the search workload will be targeted for. System index is used mostly for internal components and do mostly light weight requests. Similarly for search throttled indices the intention is to not be used for latency sensitive usecase but to keep the tps low and probably for long running requests to avoid impact on critical request path. We can fallback to non-concurrent path for these indices request.
2. Other option will be to create separate searcher threadpool for these indices as well and use those for concurrently executing the search requests on those indices

Related component

Search:Performance

Describe alternatives you've considered

No response

Additional context

No response

[jed326]

Thanks @sohami, I can take a look into this!