Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Derived field updates for 2.17 #8244

Merged
merged 9 commits into from
Sep 13, 2024
Merged

Derived field updates for 2.17 #8244

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2ea5e80
update aggregation support in limitations and add sample req/response.
mch2 Sep 11, 2024
21ab396
remove concurrent search limitation
mch2 Sep 11, 2024
57d4bf1
minor updates
mch2 Sep 11, 2024
a993bb9
fix style job error
mch2 Sep 11, 2024
e51dcb5
Merge branch 'main' into derived
mch2 Sep 12, 2024
95173fb
Doc review
kolchfa-aws Sep 12, 2024
2c6eab0
Clarify the list of unsupported geo aggregations
kolchfa-aws Sep 12, 2024
ec592f2
Apply suggestions from code review
kolchfa-aws Sep 13, 2024
1f69d0a
Merge branch 'main' into derived
kolchfa-aws Sep 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 76 additions & 2 deletions _field-types/supported-field-types/derived.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@

Currently, derived fields have the following limitations:

- **Aggregation, scoring, and sorting**: Not yet supported.
- **Scoring and sorting**: Not yet supported.
- **Aggregations**: Starting with OpenSearch 2.17, derived fields support most aggregation types. The following aggregations are not supported: geographic (geodistance, geohash grid, geohex grid, geotile grid, geobounds, geocentroid), significant terms, significant text, and scripted metric.

Check failure on line 32 in _field-types/supported-field-types/derived.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: geocentroid. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: geocentroid. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_field-types/supported-field-types/derived.md", "range": {"start": {"line": 32, "column": 222}}}, "severity": "ERROR"}
- **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name.
- **Chained derived fields**: One derived field cannot be used to define another derived field.
- **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/).
- **Concurrent segment search**: Derived fields are not supported for [concurrent segment search]({{site.url}}{{site.baseurl}}/search-plugins/concurrent-segment-search/).

We are planning to address these limitations in future versions.

Expand Down Expand Up @@ -541,6 +541,80 @@
```
</details>

## Aggregations

Starting with OpenSearch 2.17, derived fields support most aggregation types.

Geographic, significant terms, significant text, and scripted metric aggregations are not supported.
{: .note}

For example, the following request creates a simple `terms` aggregation on the `method` derived field:

```json
POST /logs/_search
{
"size": 0,
"aggs": {
"methods": {
"terms": {
"field": "method"
}
}
}
}
```
{% include copy-curl.html %}

The response contains the following buckets:

<details markdown="block">
<summary>
Response
</summary>
{: .text-delta}

```json
{
"took" : 12,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"methods" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "GET",
"doc_count" : 2
},
{
"key" : "POST",
"doc_count" : 2
},
{
"key" : "DELETE",
"doc_count" : 1
}
]
}
}
}
```
</details>

## Performance

Derived fields are not indexed but are computed dynamically by retrieving values from the `_source` field or doc values. Thus, they run more slowly. To improve performance, try the following:
Expand Down
Loading