[BUG] Since OpenSearch 2.12, when using flat_object, the error Preview of field's value: 'null' occurs.

[naomichi-y]

Describe the bug

I'm using flat_object with OpenSearch.
In OpenSearch 2.12 or later, an error occurs if the field name to be registered in flat_object has more than 10 digits.

Related component

Indexing

To Reproduce

1. Register an index template.

curl -X PUT "http://localhost:9200/_index_template/test" \
-H "Content-Type: application/json" \
-d '
{
   "index_patterns": [
      "test-*"
   ],
   "template": {
      "settings": {
         "index": {
            "number_of_shards": 1,
            "number_of_replicas": 0
         }
      },
      "mappings": {
         "properties": {
            "data": {
               "type": "object",
               "properties": {
                  "detail": {
                     "type": "flat_object"
                  }
               }
            }
         }
      }
   },
   "data_stream": {
      "timestamp_field": {
         "name": "@timestamp"
      }
   }
}
'

2. Register data in the following format.

curl -X POST "http://localhost:9200/_bulk" \
-H "Content-Type: application/x-ndjson" \
-d '
{"create":{"_index":"test-2024.04.15"}}
{"@timestamp": "2023-10-20T18:46:28Z", "data":{"detail":{"foooooooooo":[{"name":"baz"},{"name":"baz"}]}}}
'

3. The following error occurs in OpenSearch 2.12 or later.

{
   "took":3,
   "errors":true,
   "items":[
      {
         "create":{
            "_index":".ds-test-2024.04.15-000001",
            "_id":"IEAB3o4Bu0tRxJSPrze4",
            "status":400,
            "error":{
               "type":"mapper_parsing_exception",
               "reason":"failed to parse field [data.detail] of type [flat_object] in document with id 'IEAB3o4Bu0tRxJSPrze4'. Preview of field's value: 'null'",
               "caused_by":{
                  "type":"string_index_out_of_bounds_exception",
                  "reason":"String index out of range: -1"
               }
            }
         }
      }
   ]
}

I'm getting an internal error String index out of range: -1, so I tried renaming the fooooooooooo field (11) to fooooooooooo (10) and that solved the error.

Expected behavior

The error does not occur until 2.11.

{
   "took":23,
   "errors":false,
   "items":[
      {
         "create":{
            "_index":".ds-test-2024.04.15-000001",
            "_id":"wFIK3o4BVRoqI_57sTFZ",
            "_version":1,
            "result":"created",
            "_shards":{
               "total":1,
               "successful":1,
               "failed":0
            },
            "_seq_no":1,
            "_primary_term":2,
            "status":201
         }
      }
   ]
}

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context

The problem is occurring using Data Streams, but the same problem is reproduced without Data Streams.

[dblock]

@naomichi-y looks like a regression, want to turn this into a (failing) YAML REST test?

[naomichi-y]

@dblock
I have created a test case for OpenSearch, but I am not very familiar with the notation used in OpenSearch tests.
Although my attempt may be somewhat disorganized, you can reproduce the issue using the steps provided below.

"Test":
  - do:
      indices.create:
        index: test-2024.04.15
        body:
          settings:
            index:
              number_of_shards: 1
              number_of_replicas: 0
          mappings:
            properties:
              data:
                type: "object"
                properties:
                  detail:
                    type: "flat_object"

  - do:
      index:
        index: test-2024.04.15
        refresh: true
        body:
          "@timestamp": "2023-10-20T18:46:28Z"
          data:
            detail:
              fooooooooooo: [{"name":"baz"},{"name":"baz"}]

With 2.11, the results are as follows.

BUILD SUCCESSFUL in 9s

2.12 has an error.

java.lang.AssertionError: Failure at [cat.segments/10_basic:19]: expected [2xx] status code but api [index] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse field [data.detail] of type [flat_object] in document with id 'VwuZ544B7p3uWqkB79Hh'. Preview of field's value: 'null'","stack_trace":"MapperParsingException[failed to parse field [data.detail] of type [flat_object] in document with id 'VwuZ544B7p3uWqkB79Hh'.
...
BUILD FAILED in 1m 58s

The test succeeds if the field name is 10 digits.

[dblock]

@naomichi-y Thank you, this is perfect. Clearly a regression. Want to bisect it to the PR that broke it, next?

[naomichi-y]

@dblock
I have probably identified the cause. My tests pass when I revert this line to the original code.

bd5b5ee#diff-9b94311ef376a05bce2bbe32e99582f739378209a89aaa6dc4bc192e4b8f2660R111

[dblock]

Looks like this was introduced in #11425 (cc: @msfroh).

Do you know the proper way to fix this @naomichi-y? Go for it!

[naomichi-y]

Okay. I'm getting a StringIndexOutOfBoundsException, so the following code might solve it. I'll look into it a bit more.

if (dotIndex != -1) {
    path.setLength(Math.max(0, path.length() - currentFieldName.length() - 1));
}

[dblock]

Start by writing a unit test in server/src/test/java/org/opensearch/common/xcontent/JsonToStringXContentParserTests.java that hits this problem. What is the value of path that needs to be passed in to fail this way?

[djmadeira]

Was this ever released? I see the fix for it was merged but I'm still hitting this on 2.14.0

Edit: My bad I was pulling the wrong docker tag

[dblock]

Fixed in #13259. Closing.