[BUG] Removing _index field by remove ingest processor causes OpenSearch to crash

[gaobinlong]

Describe the bug
When removing the _index field by remove ingest processor, the OpenSearch cluster crashes with a failed assertion.

To Reproduce
Steps to reproduce the behavior:

1. Create a ingest pipeline

PUT _ingest/pipeline/test1
{
  "processors": [
      {
        "remove": {
          "field": "_index"
        }
      }
    ]
}

2. Add a doc

POST test2/_doc/1?pipeline=test1
{
   "foo1": "bar"
}

you can see that OpenSearch cluster crashed with the following exception:

[2023-10-19T11:11:14,602][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] fatal error in thread [opensearch[node-1][write][T#2]], exiting
java.lang.AssertionError: null
	at org.opensearch.ingest.IngestService.executePipelines(IngestService.java:671) ~[classes/:?]
	at org.opensearch.ingest.IngestService$3.doRun(IngestService.java:560) ~[classes/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[classes/:?]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[classes/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
fatal error in thread [opensearch[node-1][write][T#2]], exiting

Expected behavior
OpenSearch can handle this case correctly, the metadata fields such as _index, _id and others could not be removed by remove ingest processor.

Host/Environment (please complete the following information):

• OS: [macOS]
• Version [2.9]

[gaobinlong]

Removing _version field causes null_pointer_exception:
Request:

PUT _ingest/pipeline/test1
{
  "processors": [
      {
        "remove": {
          "field": "_version"
        }
      }
    ]
}
POST test2/_doc/1?pipeline=test1
{
   "a":"x1"
}

Response:

{
  "error": {
    "root_cause": [
      {
        "type": "null_pointer_exception",
        "reason": "Cannot invoke \"java.lang.Number.longValue()\" because the return value of \"java.util.Map.get(Object)\" is null"
      }
    ],
    "type": "null_pointer_exception",
    "reason": "Cannot invoke \"java.lang.Number.longValue()\" because the return value of \"java.util.Map.get(Object)\" is null"
  },
  "status": 500
}

[gaobinlong]

In most time remove _id by remove ingest processor works well, but if we pass version and version_type parameters when calling index API, removing _id will also cause OpenSearch to crash.

PUT _ingest/pipeline/test
{
  "processors": [
      {
        "remove": {
          "field": "_id"
        }
      }
    ]
}
POST test_123/_doc/1?version=1&version_type=external&pipeline=test2
{
  "foo.x":1
}

, then you can see the following error:

[2023-10-24T16:01:40,373][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] fatal error in thread [opensearch[node-1][write][T#4]], exiting
java.lang.AssertionError: version: 1 type: EXTERNAL
	at org.opensearch.index.engine.InternalEngine.assertPrimaryCanOptimizeAddDocument(InternalEngine.java:801) ~[classes/:?]
	at org.opensearch.index.engine.InternalEngine.canOptimizeAddDocument(InternalEngine.java:779) ~[classes/:?]
	at org.opensearch.index.engine.InternalEngine.planIndexingAsPrimary(InternalEngine.java:1041) ~[classes/:?]
	at org.opensearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1029) ~[classes/:?]
	at org.opensearch.index.engine.InternalEngine.index(InternalEngine.java:883) ~[classes/:?]
	at org.opensearch.index.shard.IndexShard.index(IndexShard.java:1118) ~[classes/:?]
	at org.opensearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1063) ~[classes/:?]
	at org.opensearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:954) ~[classes/:?]
	at org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:622) ~[classes/:?]
	at org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:469) ~[classes/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:908) ~[classes/:?]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[classes/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]

[gaobinlong]

Closed by #10895.