[BUG]For ES version after 6.0.0 the index seems to be created with " index.mapper.dynamic " attribute which ideally it should not have

[niravpi]

Describe the bug
For ES version after 6.0.0 seen some specific case where the index seems to be created with " index.mapper.dynamic " attribute which should not happened as while I am trying the change this setting getting below error

"error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Setting index.mapper.dynamic was removed after version 6.0.0"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "Setting index.mapper.dynamic was removed after version 6.0.0"
  },
  "status" : 400

But where as I can see that some index which are created in the 1.0.0 version are having this settings which we are not able to change or remove

Sample settings

"index" : {
        "creation_date_string" : "2022-03-03T00:45:30.711Z",
        "refresh_interval" : "30s",
        "number_of_shards" : "5",
        "provided_name" : "index name",
        "mapper" : {
          "dynamic" : "false"
        },
        "creation_date" : "1646268330711",
        "number_of_replicas" : "3",
        "uuid" : "Some UUID",
        "version" : {
          "created_string" : "1.0.0",
          "created" : "135217827"
        }
      }

Expected behavior
Newly created index on ES version above 6.0.0 should not have this settings

[dblock]

Is this still the case in 2.x? Do you have repro steps, please?

[shwetathareja]

@dblock : this code path has not changed in OpenSearch 2.0, code reference It might still be there. I will take a look at this one.

[derek-ho]

Hey folks, I am trying to connect Jaeger with Opensearch and getting this same error on startup of the Jaeger Collector. Could somebody take a look? Happy to hop on a call and try to reproduce this issue @dblock @shwetathareja

[shwetathareja]

@derek-ho : can you provide details on Jaeger Collector? which version are you using? How does it connect? Is the index created manually or using auto creation? is the error for existing index or during creation?

[derek-ho]

I am using Jaeger version 1.38.1. I believe it is connecting with the ES go client:
https://github.com/jaegertracing/jaeger/blob/3d67ac86f53866332568b0f542265b00d86407fc/plugin/storage/es/spanstore/writer.go#L48
https://github.com/jaegertracing/jaeger/blob/3d67ac86f53866332568b0f542265b00d86407fc/plugin/storage/es/spanstore/writer.go#L110
But I am not sure. I think it is getting auto created, and the error seems to imply it is for creating the index itself. Let's jump on a call next week to dig deeper.

[derek-ho]

They do not have a dedicated Opensearch connection package yet - this is using the same one that is used to connect to Elasticsearch; that is one of the TODO's. However, I would like to investigate whether this problem might arise when a Opensearch connection package is written as well.

[shutlle]

Hi, we have same error with OpenSearch 2.2.0 and Jaeger 1.38.1
Any additional information about this issue?

[derek-ho]

@shutlle can you run with --es.version=7? That fixed the issue for me.

[hardened-user]

@derek-ho it's work

[peternied]

Saw this issue today, I'll see about making this more addressable so that you can 'unset' the value rather than being stuck deleting the index.

e.g the following would be allowed

PUT /index-1/_settings
{
   "index": {
      "mapper.dynamic": null
    }
}

[peternied]

See #11193 for the fix, I ended up expanding the fix to allow the setting value to be null or false - this should mitigate migration issues when no backwards capability issue would occur. This would also allow indices that were using this setting to be unset so they can be cleanly migrated.

[mgodwan]

@peternied Do we know the root cause around how this setting was added to the index in OS 1.0 created index in the first place? I was trying to go through the issue but could not find the root cause through the trail. Is it just part of include_defaults response?

[peternied]

root cause around how this setting was added to the index in OS 1.0 created index in the first place

@mgodwan No, in the cases where I've encountered this issue the cluster operator did not remember the historical context and I didn't find any traces of how the index came to be. I'm going to close this specific issue as I've got a fix for this case, lets keep tracking the root cause investigation in [1].

• [1] [BUG] IndexSettings gets into invalid states on a cluster #12350