AD Enhancements in Version 2.15 (opensearch-project#7388)

* Enhancements in Version 2.15 Starting from version 2.15, we have introduced several enhancements: 1. Custom Index Management: * Added support for custom index management. For more details, watch the video on the create detector page and detector detail page anomaly-detection-dashboards-plugin#770. * Custom result indices are now managed as aliases. Consequently, additional security permissions are required for management. 2. New JVM Heap Usage Threshold Setting: * Introduced a new setting, plugins.anomaly_detection.jvm_heap_usage_threshold, to manage the memory circuit breaker threshold. 3. Documentation Improvements: * Added examples for DSL filters to enhance the documentation. 4. Ruby Version Update: * Updated Ruby version in CONTRIBUTING.md from 3.2 to 3.2.4 as version 3.2 is not available. Testing done: * built and viewed the changed website locally Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/settings.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/settings.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Kaituo Li <[email protected]> * Update _observing-your-data/ad/index.md Signed-off-by: Melissa Vagi <[email protected]> * Update _observing-your-data/ad/index.md Signed-off-by: Melissa Vagi <[email protected]> * Update index.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _observing-your-data/ad/settings.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> --------- Signed-off-by: Kaituo Li <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
jed326 · Jun 17, 2024 · f3153d8 · f3153d8
1 parent 8efaf3e
commit f3153d8
Show file tree

Hide file tree

Showing 3 changed files with 57 additions and 14 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -82,7 +82,7 @@ Follow these steps to set up your local copy of the repository:
 
    ```
    curl -sSL https://get.rvm.io | bash -s stable
-   rvm install 3.2
+   rvm install 3.2.4
    ruby -v
    ```
 

diff --git a/_observing-your-data/ad/index.md b/_observing-your-data/ad/index.md
@@ -30,7 +30,39 @@ A detector is an individual anomaly detection task. You can define multiple dete
    - Enter a name and brief description. Make sure the name is unique and descriptive enough to help you to identify the purpose of the detector.
 1. Specify the data source.   
    - For **Data source**, choose the index you want to use as the data source. You can optionally use index patterns to choose multiple indexes.
-   - (Optional) For **Data filter**, filter the index you chose as the data source. From the **Data filter** menu, choose **Add data filter**, and then design your filter query by selecting **Field**, **Operator**, and **Value**, or choose **Use query DSL** and add your own JSON filter query.
+   - (Optional) For **Data filter**, filter the index you chose as the data source. From the **Data filter** menu, choose **Add data filter**, and then design your filter query by selecting **Field**, **Operator**, and **Value**, or choose **Use query DSL** and add your own JSON filter query. Only [Boolean queries]({{site.url}}{{site.baseurl}}/query-dsl/compound/bool/) are supported for query domain-specific language (DSL).
+
+#### Example filter using query DSL
+The query is designed to retrieve documents in which the `urlPath.keyword` field matches one of the following specified values:
+
+   - /domain/{id}/short
+   - /sub_dir/{id}/short
+   - /abcd/123/{id}/xyz
+
+   ```json
+   {
+      "bool": {
+         "should": [
+               {
+                  "term": {
+                     "urlPath.keyword": "/domain/{id}/short"
+                  }
+               },
+               {
+                  "term": {
+                     "urlPath.keyword": "/sub_dir/{id}/short"
+                  }
+               },
+               {
+                  "term": {
+                     "urlPath.keyword": "/abcd/123/{id}/xyz"
+                  }
+               }
+         ]
+      }
+   }
+   ```
+
 1. Specify a timestamp.    
    - Select the **Timestamp field** in your index.
 1. Define operation settings.
@@ -44,23 +76,33 @@ A detector is an individual anomaly detection task. You can define multiple dete
    - (Optional) To add extra processing time for data collection, specify a **Window delay** value.
       - This value tells the detector that the data is not ingested into OpenSearch in real time but with a certain delay. Set the window delay to shift the detector interval to account for this delay.
       - For example, say the detector interval is 10 minutes and data is ingested into your cluster with a general delay of 1 minute. Assume the detector runs at 2:00. The detector attempts to get the last 10 minutes of data from 1:50 to 2:00, but because of the 1-minute delay, it only gets 9 minutes of data and misses the data from 1:59 to 2:00. Setting the window delay to 1 minute shifts the interval window to 1:49--1:59, so the detector accounts for all 10 minutes of the detector interval time.
-1. Specify custom result index.
-   - If you want to store the anomaly detection results in your own index, choose **Enable custom result index** and specify the custom index to store the result. The anomaly detection plugin adds an `opensearch-ad-plugin-result-` prefix to the index name that you input. For example, if you input `abc` as the result index name, the final index name is `opensearch-ad-plugin-result-abc`.
+1. Specify custom results index.
+   - The Anomaly Detection plugin allows you to store anomaly detection results in a custom index of your choice. To enable this, select **Enable custom results index** and provide a name for your index, for example, `abc`. The plugin then creates an alias prefixed with `opensearch-ad-plugin-result-` followed by your chosen name, for example, `opensearch-ad-plugin-result-abc`. This alias points to an actual index with a name containing the date and a sequence number, like `opensearch-ad-plugin-result-abc-history-2024.06.12-000002`, where your results are stored.
 
-   You can use the dash “-” sign to separate the namespace to manage custom result index permissions. For example, if you use `opensearch-ad-plugin-result-financial-us-group1` as the result index, you can create a permission role based on the pattern `opensearch-ad-plugin-result-financial-us-*` to represent the "financial" department at a granular level for the "us" area.
+   You can use the dash “-” sign to separate the namespace to manage custom results index permissions. For example, if you use `opensearch-ad-plugin-result-financial-us-group1` as the results index, you can create a permission role based on the pattern `opensearch-ad-plugin-result-financial-us-*` to represent the "financial" department at a granular level for the "us" area.
    {: .note }
 
-      - If the custom index you specify doesn’t already exist, the Anomaly Detection plugin creates this index when you create the detector and start your real-time or historical analysis.
-      - If the custom index already exists, the plugin checks if the index mapping of the custom index matches the anomaly result file. You need to make sure the custom index has valid mapping as shown here: [anomaly-results.json](https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/resources/mappings/anomaly-results.json).
-   - To use the custom result index option, you need the following permissions:
-      - `indices:admin/create` - If the custom index already exists, you don't need this.
+   - When the Security plugin (fine-grained access control) is enabled, the default results index becomes a system index and is no longer accessible through the standard Index or Search APIs. To access its content, you must use the Anomaly Detection RESTful API or the dashboard. As a result, you cannot build customized dashboards using the default results index if the Security plugin is enabled. However, you can create a custom results index in order to build customized dashboards.
+   - If the custom index you specify does not exist, the Anomaly Detection plugin will create it when you create the detector and start your real-time or historical analysis.
+   - If the custom index already exists, the plugin will verify that the index mapping matches the required structure for anomaly results. In this case, ensure that the custom index has a valid mapping as defined in the [`anomaly-results.json`](https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/resources/mappings/anomaly-results.json) file.
+   - To use the custom results index option, you need the following permissions:
+      - `indices:admin/create` - The Anomaly Detection plugin requires the ability to create and roll over the custom index.
+      - `indices:admin/aliases` - The Anomaly Detection plugin requires access to create and manage an alias for the custom index.
       - `indices:data/write/index` - You need the `write` permission for the Anomaly Detection plugin to write results into the custom index for a single-entity detector.
-      - `indices:data/read/search` - You need the `search` permission because the Anomaly Detection plugin needs to search custom result indexes to show results on the anomaly detection UI.
+      - `indices:data/read/search` - You need the `search` permission because the Anomaly Detection plugin needs to search custom results indexes to show results on the Anomaly Detection UI.
       - `indices:data/write/delete` - Because the detector might generate a large number of anomaly results, you need the `delete` permission to delete old data and save disk space.
       - `indices:data/write/bulk*` -  You need the `bulk*` permission because the Anomaly Detection plugin uses the bulk API to write results into the custom index.
-   - Managing the custom result index:
-      - The anomaly detection dashboard queries all detectors’ results from all custom result indexes. Having too many custom result indexes might impact the performance of the Anomaly Detection plugin.
-      - You can use [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/ism/index/) to rollover old result indexes. You can also manually delete or archive any old result indexes. We recommend reusing a custom result index for multiple detectors.
+   - Managing the custom results index:
+      - The anomaly detection dashboard queries all detectors’ results from all custom results indexes. Having too many custom results indexes might impact the performance of the Anomaly Detection plugin.
+      - You can use [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/ism/index/) to rollover old results indexes. You can also manually delete or archive any old results indexes. We recommend reusing a custom results index for multiple detectors.
+      - The Anomaly Detection plugin also provides lifecycle management for custom indexes. It rolls an alias over to a new index when the custom results index meets any of the conditions in the following table.
+
+      Parameter | Description | Type | Unit | Example | Required
+      :--- | :--- |:--- |:--- |:--- |:---
+      `result_index_min_size` | The minimum total primary shard size (excluding replicas) required for index rollover. If set to 100 GiB and the index has 5 primary and 5 replica shards of 20 GiB each, then the total primary shard size is 100 GiB, triggering the rollover. | `integer` | `MB` | `51200` | No
+      `result_index_min_age` |  The minimum index age required for rollover, calculated from its creation time to the current time. | `integer` |`day` | `7` | No
+      `result_index_ttl` | The minimum age required to permanently delete rolled-over indexes. | `integer` | `day` | `60` | No
+
 1. Choose **Next**.   
 
 After you define the detector, the next step is to configure the model.

diff --git a/_observing-your-data/ad/settings.md b/_observing-your-data/ad/settings.md
@@ -49,4 +49,5 @@ plugins.anomaly_detection.dedicated_cache_size | 10 | If the real-time analysis
 plugins.anomaly_detection.max_concurrent_preview | 2 | The maximum number of concurrent previews. You can use this setting to limit resource usage.
 plugins.anomaly_detection.model_max_size_percent | 0.1 | The upper bound of the memory percentage for a model.
 plugins.anomaly_detection.door_keeper_in_cache.enabled | False | When set to `true`, OpenSearch places a bloom filter in front of an inactive entity cache to filter out items that are not likely to appear more than once.
-plugins.anomaly_detection.hcad_cold_start_interpolation.enabled | False | When set to `true`, enables interpolation in high-cardinality anomaly detection (HCAD) cold start.
+plugins.anomaly_detection.hcad_cold_start_interpolation.enabled | False | When set to true, enables interpolation for high-cardinality anomaly detection (HCAD) during the initial cold start period.
+plugins.anomaly_detection.jvm_heap_usage_threshold | 95 | Specifies the JVM memory usage threshold, as a percentage, at which anomaly detectors will be disabled. The default value is 95%, meaning that detectors will be disabled when JVM heap usage reaches 95%.
-Original file line number
+Diff line change
@@ Expand Up @@
        ```
        curl -sSL https://get.rvm.io | bash -s stable
-       rvm install 3.2
+       rvm install 3.2.4
        ruby -v
        ```
@@ Expand Down @@