[BUG] regression in 2.17: error log on each request when multiple JWT auth domains are configured

[rursprung]

What is the bug?
if a cluster has two (or more) JWT auth domains specified and a request is issued which is signed by the 2nd (or 3rd, etc.) issuer and issuer verification is enabled (i didn't test without, to be honest) the following message is written to the log:

{"type": "server", "timestamp": "2024-11-15T18:00:57,045Z", "level": "ERROR", "component": "c.a.d.a.h.j.HTTPJwtAuthenticator", "cluster.name": "test-opensearch", "node.name": "test-opensearch-0", "message": "Failed to parse JWT token using any of the available parsers", "cluster.uuid": "EVQZugEsRp2_FQw9l1sVFA", "node.id": "94OoEgHIQluhdUBPRJeP8g" }

the request itself still succeeds.

enabling debug logging gives more information:

{"type": "server", "timestamp": "2024-11-15T17:52:45,380Z", "level": "DEBUG", "component": "o.o.s.a.BackendRegistry", "cluster.name": "test-opensearch", "node.name": "test-opensearch-0", "message": "Check authdomain for rest noop/0 or 2 in total", "cluster.uuid": "EVQZugEsRp2_FQw9l1sVFA", "node.id": "94OoEgHIQluhdUBPRJeP8g" }
{"type": "server", "timestamp": "2024-11-15T17:52:45,380Z", "level": "DEBUG", "component": "c.a.d.a.h.j.HTTPJwtAuthenticator", "cluster.name": "test-opensearch", "node.name": "test-opensearch-0", "message": "Invalid or expired JWT token.", "cluster.uuid": "EVQZugEsRp2_FQw9l1sVFA", "node.id": "94OoEgHIQluhdUBPRJeP8g" ,
"stacktrace": ["io.jsonwebtoken.IncorrectClaimException: Expected iss claim to be: ISS0, but was: ISS1.",
"at io.jsonwebtoken.impl.DefaultJwtParser.validateExpectedClaims(DefaultJwtParser.java:773) ~[jjwt-impl-0.12.6.jar:0.12.6]",
[..]
"at com.amazon.dlic.auth.http.jwt.HTTPJwtAuthenticator.extractCredentials(HTTPJwtAuthenticator.java:121) [opensearch-security-2.17.1.0.jar:2.17.1.0]",
"at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:279) [opensearch-security-2.17.1.0.jar:2.17.1.0]",
"at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:308) [opensearch-security-2.17.1.0.jar:2.17.1.0]",
[..]
{"type": "server", "timestamp": "2024-11-15T17:52:45,381Z", "level": "ERROR", "component": "c.a.d.a.h.j.HTTPJwtAuthenticator", "cluster.name": "test-opensearch", "node.name": "test-opensearch-0", "message": "Failed to parse JWT token using any of the available parsers", "cluster.uuid": "EVQZugEsRp2_FQw9l1sVFA", "node.id": "94OoEgHIQluhdUBPRJeP8g" }
{"type": "server", "timestamp": "2024-11-15T17:52:45,381Z", "level": "DEBUG", "component": "o.o.s.a.BackendRegistry", "cluster.name": "test-opensearch", "node.name": "test-opensearch-0", "message": "Check authdomain for rest noop/1 or 2 in total", "cluster.uuid": "EVQZugEsRp2_FQw9l1sVFA", "node.id": "94OoEgHIQluhdUBPRJeP8g" }
{"type": "server", "timestamp": "2024-11-15T17:52:45,381Z", "level": "DEBUG", "component": "o.o.s.a.BackendRegistry", "cluster.name": "test-opensearch", "node.name": "test-opensearch-0", "message": "Rest user 'User [name=testuser, backend_roles=[testrole], requestedTenant=null]' is authenticated", "cluster.uuid": "EVQZugEsRp2_FQw9l1sVFA", "node.id": "94OoEgHIQluhdUBPRJeP8g" }

How can one reproduce the bug?

1. set up an OpenSearch cluster with two JWT auth domains (with different issuers) and set required_issuer and required_audience for both of them
2. create a JWT for the second issuer (based on the order of the auth domain)
3. run any request against the cluster with this JWT, e.g. curl -H "Authorization: Bearer ${JWT}" http://localhost:9200/_plugins/_security/authinfo
4. check the log => error is in there but the request succeeded
5. re-try with a JWT for the first issuer => no log entry (and request succeeds, of course)

An auth domain would look something like this:

                "jwt_auth_domain_0": {
                    "http_enabled": true,
                    "order": 0,
                    "http_authenticator": {
                        "challenge": false,
                        "type": "jwt",
                        "config": {
                            "signing_key": "your public key here",
                            "jwt_header": "Authorization",
                            "roles_key": "your role key here",
                            "required_issuer": "your issuer name here",
                            "required_audience": "your audience name here"
                        }
                    },
                    "authentication_backend": {
                        "type": "noop",
                        "config": {}
                    },
                    "description": "Authenticate via Json Web Token 0"
                }

What is the expected behavior?
no error in the log if authentication worked. this error makes operations hard as there are many error entries and it's unclear whether these are real errors or false positives.

What is your host/environment?

• OS: linux
• Version: 2.17.1
• Plugins: security + some unrelated plugins

Do you have any screenshots?
n/a

Do you have any additional context?
this regression seems to have been introduced with 6c431e1 (#4632), CC @stephen-crawford

it seems that this code change was written under the assumption that there is only one single auth domain, in which case the error would make sense. but if there are multiple auth domains it's normal that it would fail for all but one (which may or may not be the first).

[cwperks]

[Triage] Thank you for filing this issue @rursprung. I raised a PR to lower the log level of this statement.