THREESCALE-10224 CVE-2023-44487 http/2 rapid reset

[eguzki]

What

Fixes https://issues.redhat.com/browse/THREESCALE-10224

CVE-2023-44487 (HTTP/2 Rapid Reset) fix

The actual fix is two-fold:

• Bumping openresty to 1.19.3-23. The upgrade brings the fix from nginx http://hg.nginx.org/nginx/rev/cdda286c0f1b
• Enable openresty [lua_check_client_abort|https://github.com/openresty/lua-nginx-module#lua_check_client_abort] to check for premature client connection abortion.

TL;DR the Rapid reset sec threat is about doing lot's of requests with rapid reset right after sending the request so the [http2_max_concurrent_streams|http://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_concurrent_streams] threshold is never reached.

With the openresty [lua_check_client_abort|https://github.com/openresty/lua-nginx-module#lua_check_client_abort] disabled, apicast was not honoring the resets and the http2_max_concurrent_streams threshold (128 by default in nginx) was hit. Even though the client was sending the reset. So, APICast was protected to the rapid reset sec threat. However that happened because HTTP2 protocol was not correctly implemented. APICast was sending data even after receiving the reset in the closed stream.

Thus, this PR enables lua_check_client_abort, which makes APIcast to respect the resets, thus making it vulnerable to the rapid reset sec threat. But also more compliant with HTTP2. The protection against the rapid reset sec threat comes from the nginx fix http://hg.nginx.org/nginx/rev/cdda286c0f1b

The client used to exploit the rapid reset vulnerability: https://github.com/eguzki/nice-cve-poc/tree/ssl-optional

Related links:

• https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/
• https://www.nginx.com/blog/http-2-rapid-reset-attack-impacting-f5-nginx-products/

Verification steps

• Checkout this branch

git checkout origin/cve-2023-44487-rapid-reset

• Build docker image apicast-cve-2023-44487

make runtime-image IMAGE_NAME=apicast-cve-2023-44487

• Run dummy HTTP/2 echo api

docker run --name echoserver --rm -p 8002-8003:8002-8003 kalmhq/echoserver:latest

• Create gateway certs for example.com (inside the container)

cd /tmp/apicast-test && cd /tmp/apicast-test
openssl genrsa -out rootCA.key 2048
openssl req -batch -new -x509 -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem
openssl req -subj '/CN=example.com'  -newkey rsa:4096 -nodes -sha256 -days 3650 -keyout example.com.key -out example.com.csr
chmod +r example.com.key
openssl x509 -req -in example.com.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out example.com.crt -days 500 -sha256

• Create apicast configuration

ECHOSERVER_IP=$(docker inspect echoserver | yq e -P '.[0].NetworkSettings.Networks.bridge.IPAddress' -)
❯ cat <<EOF >config.json
{
  "services": [
    {
      "backend_version": "1",
      "proxy": {
        "hosts": [
          "example.com"
        ],
        "api_backend": "https://${ECHOSERVER_IP}:8003",
        "backend": {
          "endpoint": "http://127.0.0.1:8081",
          "host": "backend"
        },
        "proxy_rules": [
          {
            "http_method": "GET",
            "pattern": "/",
            "metric_system_name": "hits",
            "delta": 1,
            "parameters": [],
            "querystring_parameters": {}
          }
        ],
        "policy_chain": [
          {
                "name": "default_credentials",
                "version": "builtin",
                "configuration": {
                  "auth_type": "user_key",
                  "user_key": "some_dummy_key"
                }
          },
          {
            "name": "apicast.policy.apicast"
          }
        ]
      }
    }
  ]
}
EOF

• Run apicast locally. the output will be filtered, looking for the log client sent too many streams at once while processing HTTP/2 connection. That log is generated by the fix from nginx http://hg.nginx.org/nginx/rev/cdda286c0f1b

docker run -t --name apicast --rm -v $PWD:/var/run/secrets/apicast -v $PWD/config.json:/opt/app/config.json:ro --env THREESCALE_CONFIG_FILE=/opt/app/config.json --env APICAST_LOG_LEVEL=info --env APICAST_CONFIGURATION_LOADER=lazy --env THREESCALE_DEPLOYMENT_ENV=staging --env APICAST_HTTPS_PORT=8443 --env APICAST_HTTPS_CERTIFICATE=/var/run/secrets/apicast/example.com.crt --env APICAST_HTTPS_CERTIFICATE_KEY=/var/run/secrets/apicast/example.com.key --env APICAST_WORKERS=1 --env APICAST_CONFIGURATION_CACHE=0 apicast-cve-2023-44487 | grep "client sent too many streams at once while processing HTTP/2 connection"

• Test rapid reset traffic generator from https://github.com/eguzki/nice-cve-poc/tree/ssl-optional

APICAST_IP=$(docker inspect apicast | yq e -P '.[0].NetworkSettings.Networks.bridge.IPAddress' -)

# Just one request to test e2e works
curl --http2 --resolve example.com:8443:${APICAST_IP} -v --cacert rootCA.pem "https://example.com:8443"

# Run the traffic tool. The following run sends 300 request/reset frames. Depending on your resources, more or less is required. At least 2 * max_concurrent_streams (which is 128 in nginx by default)
docker run --rm -t  quay.io/eastizle/cve-2023-44487:v0.0.2 --host example.com --hits 300  https://${APICAST_IP}:8443

The apicast log line should show

2023/10/31 17:21:03 [info] 19#19: *404 client sent too many streams at once while processing HTTP/2 connection, client: 172.17.0.4, server: 0.0.0.0:8443

[tkan145]

Look good to me. +1

Verification

• Checkout this branch

git checkout origin/cve-2023-44487-rapid-reset

• Run docker dev image

make development

• Check the running image

 ▲ lib/3scale/APIcast docker ps
CONTAINER ID   IMAGE                                           COMMAND                  CREATED         STATUS         PORTS      NAMES
487d71673b69   quay.io/3scale/apicast-ci:openresty-1.19.3-23   "cat"                    8 minutes ago   Up 8 minutes   8080/tcp   apicast_build_0-development-1
52b7c5009bc4   redis                                           "docker-entrypoint.s…"   44 hours ago    Up 8 minutes   6379/tcp   apicast_build_0-redis-1