[BUG] Invalid json error responses

[sebmjoll]

Describe the bug

When running queries against an AWS-managed OpenSearch 1.3 cluster, when the cluster is under load, we sometimes get the following error payload when trying to index a document:

{
  "message": "No server available to handle the request",
}

The trailing comma makes it invalid json and the library "@opensearch-project/opensearch" version 2.5.0 fails to parse it with errors like this:

Unexpected token } in JSON at position 60

We also get sometimes another invalid json response when searching:

{
  "message": "Empty or invalid response",
}

Related component

Other

To Reproduce

I don't know how to force errors like that.

Expected behavior

If the APIs are supposed to return strictly valid JSON, then they should not return invalid JSON.

If on the other hand, these response payloads are considered legit, the javascript "@opensearch-project/opensearch" library should not fail to parse them.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[dblock]

This seems like an AWS-specific problem that's generated on some gateway and not inside OpenSearch, or at least I couldn't find this string anywhere in this project.

Have you opened a ticket with AWS yet? If not, could you please do that? I'd like to close it here once you did.

[sebmjoll]

We are reporting this to AWS, yes.

[dblock]

Thanks @sebmjoll, I'm going to close this here. Feel free to reopen/comment if you don't get traction or a response and I can see if I can help.