Add system param to /_cat/indices to only list system indices from SystemIndexPlugin.getSystemIndexDescriptors

[cwperks]

Description

This PR adds a param to /_cat/indices called system (i.e. /_cat/indices?system=true) which allows the caller to have only the system indices printed when calling the cat indices API. The response would include all indices that plugins register via the SystemIndexPlugin.getSystemIndexDescriptors extension point.

System indices are a source of confusion and there are 2 conflicting definitions of a system index that should ultimately be unified.

1. Indices that plugins register with a plain text descriptor via the SystemIndexPlugin.getSystemIndexDescriptors extension point
2. System indices tracked by the security plugin that are given special protections by the security plugin. The security plugin protects system indices by preventing any regular user from meddling with a system index directly. Only plugins operating in a trusted mode or a user connecting with admin certificate can meddle directly with a system index.

While working with system indices, I have noticed that there is no API that can list all indices registered with SystemIndexPlugin.getSystemIndexDescriptors so I am raising a PR to modify /_cat/indices to add the functionality.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 7ad8bc7

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

[github-actions]

❌ Gradle check result for 4af2dd5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8b35852: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 7ad8bc7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[opensearch-trigger-bot]

This PR is stalled because it has been open for 30 days with no activity.

[peternied]

@cwperks Thanks for creating this PR, but I don't think this is the right kind of change. AFAIK OpenSearch doesn't expose the notion of system index in any of the other APIs so only exposing it via a filtering parameter on cat indices it.

To move forward with this change can you map out all the externally facing places that should be updated and this change would be part of that whole. IMO the first incremental change should be to return the property as part of the index settings then plumb it elsewhere.

I'm going to close this PR, after creating an issue that lays out how this would work as a whole feel free to reopen.

[cwperks]

I will draft up an issue for this to solicit comments. There's been a lot of discussion around system indices and I was surprised to find that there wasn't any way to list out all system indices and their descriptors that are defined by SystemIndexPlugin.getSystemIndexDescriptor. I think there should be some sort of API where its possible to list them out and where users can see the descriptions w/o perusing the code of each plugin/module that defines them.