[THREESCALE-10278] upgrade lua-resty-http to 0.17.1

[tkan145]

What

Fix https://issues.redhat.com/browse/THREESCALE-10278

lua-resty-http 0.17.1 is also required for https://issues.redhat.com/browse/THREESCALE-5105

This PR is mainly a refactoring of existing code so no additional integration tests/unittests are added.

Verification Steps

1. Connect via proxy

• Build docker image from this git branch

make runtime-image IMAGE_NAME=apicast-test

• Run proxy dev environment

cd dev-environments/https-proxy-upstream-tlsv1.3
make certs
make gateway IMAGE_NAME=apicast-test

• Send request to APIcast

curl --resolve get.example.com:8080:127.0.0.1 -v "http://get.example.com:8080/?user_key=123"

Request should return 200

* Added get.example.com:8080:127.0.0.1 to DNS cache
* Hostname get.example.com was found in DNS cache
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to get.example.com (127.0.0.1) port 8080 (#0)
> GET /?user_key=123 HTTP/1.1
> Host: get.example.com:8080
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< Date: Tue, 30 Jan 2024 04:53:30 GMT
< Server: gunicorn/19.9.0
< 
{
  "args": {
    "user_key": "123"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Host": "example.com", 
    "User-Agent": "curl/7.61.1"
  }, 
  "origin": "172.21.0.4", 
  "url": "http://example.com/get?user_key=123"
}
* Connection #0 to host get.example.com left intact

• Proxy should receive a CONNECT method with upstream hostname not IP:PORT

docker compose -p https-proxy-upstream-tlsv13 logs -f proxy

proxy-1  | CONNECT   Jan 30 04:55:57.831 [1]: Connect (file descriptor 4): 172.21.0.5
proxy-1  | CONNECT   Jan 30 04:55:57.831 [1]: Request (file descriptor 4): CONNECT example.com:443 HTTP/1.1

2. Fetching configuration file from 3scale

• Start dev environment

make development
make dependencies

• Run APIcast locally

THREESCALE_DEPLOYMENT_ENV=staging APICAST_LOG_LEVEL=debug APICAST_WORKER=1 APICAST_CONFIGURATION_LOADER=lazy APICAST_CONFIGURATION_CACHE=0 THREESCALE_PORTAL_ENDPOINT=https://[email protected] ./bin/apicast

• Run query

# capture apicast IP
APICAST_IP=$(docker inspect apicast_build_0-development-1 | yq e -P '.[0].NetworkSettings.Networks.apicast_build_0_default.IPAddress' -)

curl -i -k -H "Host: default-product.staging.example.com:443" "http://${APICAST_IP}:8080/?user_key=<user_key>"

Replace <user_key> with the actual user key. The response should be HTTP/1.1 200 OK

HTTP/1.1 200 OK                    
Server: openresty                  
Date: Tue, 30 Jan 2024 05:08:26 GMT
Content-Type: application/json     
Content-Length: 702                
Connection: keep-alive             
x-3scale-echo-api: echo-api/1.0.3  
vary: Origin                       
x-content-type-options: nosniff    
x-envoy-upstream-service-time: 0   

From the APIcast log, the request to fetch configuration file return 200

2024/02/14 01:42:12 [debug] 763706#763706: *2 remote_v2.lua:268: proxy_configs_per_page(): proxy configs get status: 200 url: https://[email protected]/admin/api/account/proxy_configs/production.json?host=default-product.staging.example.com&page=1&per_page=500&version=latest body: {"proxy_configs":  .... }

{
    "services": [
        {
            "proxy": {
                "error_headers_auth_failed": "text\/plain; charset=us-ascii",
                "error_headers_limits_exceeded": "text\/plain; charset=us-ascii",
                "error_headers_auth_missing": "text\/plain; charset=us-ascii",
                "error_headers_no_match": "text\/plain; charset=us-ascii",
                "error_status_no_match": 404,
                "error_status_auth_failed": 403,
                "error_status_limits_exceeded": 429,
                "error_status_auth_missing": 403,
                "secret_token": "Shared_secret_sent_from_proxy_to_API_backend_09df7e84d9ba36d8",
                "hostname_rewrite": null,
                "oidc_issuer_endpoint": null,
                "jwt_claim_with_client_id": null,
                "jwt_claim_with_client_id_type": null,
                "auth_user_key": "user_key",
                "auth_app_id": "app_id",
                "auth_app_key": "app_key",
                "oauth_login_url": null,
                "proxy_rules": [
                    {
                        "http_method": "GET",
                        "pattern": "\/",
                        "delta": 1,
                        "redirect_url": null,
                        "querystring_parameters": {},
                        "position": 1,
                        "parameters": {},
                        "metric_system_name": "hits",
                        "last": false,
                        "owner_type": "Proxy"
                        ...
                    }
                ],
                "error_auth_missing": "Authentication parameters missing",
                "api_test_path": "\/",
                "api_test_success": null,
                "apicast_configuration_driven": true,
                "oidc_issuer_type": "keycloak",
                "staging_domain": "default-product.staging.example.com",
                "production_domain": "default-product.production.example.com",
                "endpoint": "https:\/\/default-product.production.example.com: 443",
                "error_limits_exceeded": "Usage limit exceeded",
                "deployed_at": null,
                "backend": {
                    "endpoint": "https:\/\/su1.3scale.net",
                    "host": "su1.3scale.net"
                },
                "error_no_match": "No Mapping Rule matched",
                "valid?": true,
                "service_backend_version": "1",
                "hosts": [
                    "default-product.production.example.com",
                    "default-product.staging.example.com"
                ],
                "error_auth_failed": "Authentication failed",
                "lock_version": 1,
                "policy_chain": [
                    {
                        "name": "apicast",
                        "configuration": {},
                        "version": "builtin"
                    }
                ],
                "endpoint_port": 443,
                "sandbox_endpoint": "https:\/\/default-product.staging.example.com: 443",
                "authentication_method": "1",
                "api_backend": "https:\/\/echo-api.3scale.net: 443",
                "credentials_location": "query",
                "hostname_rewrite_for_sandbox": "echo-api.3scale.net"
            },          
            "backend_authentication_type": "service_token",
            "description": "",
            "name": "Default Product",
            "backend_version": "1",
            "proxiable?": true,
            "system_name": "default_product",
            ...
        },
    ]
}

[eguzki]

looking good.

[eguzki]

resty-http was downgraded to 0.15 for ARM support 6a09893

Can we test that this image can be built out of this PR using ARM arch host? If you do not have available, I can ask some other member of the team to test it.

[tkan145]

So I tried to build ARM image using docker from masterbranch

 ▲ APIcast make dev-build                                               
/usr/bin/docker buildx build --platform linux/arm64 -t apicast-development:latest \
        --build-arg OPENRESTY_RPM_VERSION=1.19.3 \                                 
        --build-arg LUAROCKS_VERSION=2.3.0 \                                       
        /3scale/APIcast -f Dockerfile.devel                    

 => [ 8/16] RUN yum config-manager --add-repo http://packages.dev.3sca.net/dev_packages_3sca_net.repo                                                                                                             3.7s    
 => ERROR [ 9/16] RUN yum install -y         openresty-1.19.3         openresty-resty-1.19.3         openresty-opentelemetry-1.19.3         openresty-opentracing-1.19.3         opentracing-cpp-devel-1.3.0      6.9s    
------                                                                                                                                                                                                                    
 > [ 9/16] RUN yum install -y         openresty-1.19.3         openresty-resty-1.19.3         openresty-opentelemetry-1.19.3         openresty-opentracing-1.19.3         opentracing-cpp-devel-1.3.0         libopentraci
cpp1-1.3.0         jaegertracing-cpp-client-0.3.1-13.el8:                                                                                                                                                                 
4.901 Devel packages from 3Scale                       81 kB/s | 220 kB     00:02                                                                                                                                         
6.661 Error:                                                                                                                                                                                                              
6.661  Problem 1: conflicting requests                                                                                                                                                                                    
6.661   - package jaegertracing-cpp-client-0.3.1-13.el8.x86_64 from packages.dev.3sca.net does not have a compatible architecture                                                                                         
6.661   - nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by jaegertracing-cpp-client-0.3.1-13.el8.x86_64 from packages.dev.3sca.net                                                                          
6.661   - nothing provides libpthread.so.0(GLIBC_2.3.2)(64bit) needed by jaegertracing-cpp-client-0.3.1-13.el8.x86_64 from packages.dev.3sca.net                                                                          
6.661   - nothing provides libc.so.6(GLIBC_2.14)(64bit) needed by jaegertracing-cpp-client-0.3.1-13.el8.x86_64 from packages.dev.3sca.net                                                                                 
6.661  Problem 2: cannot install the best candidate for the job                                                                                                                                                           
6.661   - package openresty-1.19.3-23.el8.x86_64 from packages.dev.3sca.net does not have a compatible architecture                                                                                                       
6.661   - nothing provides openresty-pcre >= 8.42-1 needed by openresty-1.19.3-23.el8.x86_64 from packages.dev.3sca.net                                                                                                   
6.661   - nothing provides openresty-zlib >= 1.2.11-3 needed by openresty-1.19.3-23.el8.x86_64 from packages.dev.3sca.net                                                                                                 
6.661  Problem 3: cannot install the best candidate for the job                                                                                                                                                           
6.661   - nothing provides openresty >= 1.19.3-23.el8 needed by openresty-resty-1.19.3-23.el8.noarch from packages.dev.3sca.net                                                                                           
6.661  Problem 4: cannot install the best candidate for the job                                                                                                                                                           
6.661   - package openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net does not have a compatible architecture                                                                                         
6.661   - nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net                                                                          
6.661   - nothing provides libpthread.so.0(GLIBC_2.3.2)(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net                                                                          
6.661   - nothing provides ld-linux-x86-64.so.2()(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net                                                                                
6.661   - nothing provides ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net                                                                       
6.661   - nothing provides libm.so.6(GLIBC_2.2.5)(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net                                                                                
6.661   - nothing provides libpthread.so.0(GLIBC_2.12)(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net                                                                           
6.661  Problem 5: cannot install the best candidate for the job                                                                                                                                                           
6.661   - package openresty-opentracing-1.19.3-23.el8.x86_64 from packages.dev.3sca.net does not have a compatible architecture                                                                                           
6.661   - nothing provides libc.so.6(GLIBC_2.14)(64bit) needed by openresty-opentracing-1.19.3-23.el8.x86_64 from packages.dev.3sca.net                                                                                   
6.661   - nothing provides libopentracing.so.1()(64bit) needed by openresty-opentracing-1.19.3-23.el8.x86_64 from packages.dev.3sca.net                                                                                   
6.661  Problem 6: cannot install the best candidate for the job                                                                                                                                                           
6.661   - package opentracing-cpp-devel-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net does not have a compatible architecture                                                                                      
6.661   - nothing provides libopentracing.so.1()(64bit) needed by opentracing-cpp-devel-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net                                                                              
6.661   - nothing provides libopentracing_mocktracer.so.1()(64bit) needed by opentracing-cpp-devel-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net                                                                   
6.661   - nothing provides libopentracing-cpp1 = 1.3.0-26.el8arches needed by opentracing-cpp-devel-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net                                                                  
6.661  Problem 7: cannot install the best candidate for the job                                                                                                                                                           
6.661   - package libopentracing-cpp1-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net does not have a compatible architecture                                                                                        
6.661   - nothing provides libc.so.6(GLIBC_2.14)(64bit) needed by libopentracing-cpp1-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net                                                                                
6.661   - nothing provides ld-linux-x86-64.so.2()(64bit) needed by libopentracing-cpp1-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net                                                                               
6.661   - nothing provides ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) needed by libopentracing-cpp1-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net                                                                      
6.661   - nothing provides libdl.so.2(GLIBC_2.2.5)(64bit) needed by libopentracing-cpp1-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net                                                                              
6.662 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)                                                                                                   
------                                                                                                                                                                                                                    
Dockerfile.devel:30                                                                                                                                                                                                       

Checking http://packages.dev.3sca.net/ I don't see any aarch64 packages. Perhaps you have a better way to build the image?

[guicassolato]

I’ve tried this but the verification steps won’t work OOTB on my system, due to missing RPMs for arm64 (aarch64). The packages we need (i.e. OpenResty 1.19.3-x and related ones within the same “family”, e.g. OpenTelemetry, OpenTracing, etc) are not available for this arch, neither in the default repos, nor in http://packages.dev.3sca.net/.

Searching my notes after #1381, I found out a few interesting things. Starting with the fact that, back then, I only touched Dockerfile.devel, but never Dockerfile. This means I was able to build the devel/ci container image on darwin/arm64, but never the runtime image, which is the one we use to build for prod IIRC.

Another important piece is that apparently I failed then to build the devel/ci image while targeting linux/arm64 platform. What I have succeeded doing was building on darwin/arm64 for linux/amd64.

To not completely diminish that as 100% useless, I recon it may have helped people to boot up the dev env container and run the test suite on a MacOS with Mx chip. But, in the end, we were just working around the limitations for running an arm64 image on an arm64 platform for dev purposes, but that’s all.

Back to this PR... While still unable to make runtime-image, there was only so much I could do. Here’s a summary:

1. Smoke tests (devel/ci image)

make dev-build IMAGE=quay.io/3scale/apicast-ci:openresty-1.19.3-pr1434-amd64
make development IMAGE=quay.io/3scale/apicast-ci:openresty-1.19.3-pr1434-amd64

Then, inside the development container:

make dependencies
make busted
make prove

Result: SUCCESS

2. “Connect via Proxy” (runtime image, Proxy w/ upstream using TLSv1.3)

make runtime-build # <========= FAILED
cd dev-environments/https-proxy-upstream-tlsv1.3
make certs
make gateway
curl --resolve get.example.com:8080:127.0.0.1 -v "http://get.example.com:8080/?user_key=123"
docker compose -p https-proxy-upstream-tlsv13 logs -f proxy

Result: FAILED

3. “Fetch config from 3scale” (devel/ci image)

Inside the development container:

THREESCALE_DEPLOYMENT_ENV=staging APICAST_LOG_LEVEL=debug APICAST_WORKER=1 APICAST_CONFIGURATION_LOADER=lazy APICAST_CONFIGURATION_CACHE=0 THREESCALE_PORTAL_ENDPOINT=https://[email protected] ./bin/apicast

From the host:

APICAST_IP=$(docker inspect apicast_build_0-development-1 | yq e -P '.[0].NetworkSettings.Networks.apicast_build_0_default.IPAddress' -)
curl -i -k -H "Host: example.com:443" -H "Accept: application/json" -H "Authorization: Bearer ${ACCESS_TOKEN}" "http://${APICAST_IP}:8080/foo"

Output:

HTTP/1.1 404 Not Found
Server: openresty
Date: Thu, 01 Feb 2024 10:56:20 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive

Result: SUCCESS (?)

[tkan145]

Thanks @guicassolato, for the last test, did you replace the placeholder values with actual 3scale values? It may also fail if you are using an older version of lua-resty-http (0.15), maybe delete the lua_modules folder and rerun make dependencies?

@eguzki so I guess we are good to merge this one?

[eguzki]

The code looks good to me. Very close to be merged. Just some enhancements proposed.

[eguzki]

building on darwin/arm64 for linux/amd64

Thanks @guicassolato

That was what I was missing. I was wondering how the hell you build dev image. building on darwin/arm64 for linux/amd64 answers that.

The aim is to allow arm64 based users to develop APIcast. If they cannot build runtime image for arm64, that's unfortunate but not a blocking issue if they can still build and run amd64 images.

[eguzki]

LGTM

just one change in the CHANGELOG requested and ready to be merged.

[eguzki]

There is a failing check about codecov. It says that the tests only cover the 72% of the added code in this PR (not the overall code coverage). While we could be more restrictive about this, I tend to think that 70% of code coverage either for the patch or the project is good enough and should not be blocking the merge.

[tkan145]

codecov is a strange one. It's fine now 😅

[tkan145]

Rebased!

[eguzki]

@tkan145 the second part of the verification steps cannot be run. You did not specify the 3scale configuration fetched from 3scale API, hence I cannot reproduce.

Nevertheless, I consider the first part good enough. And together with the (old) e2e tests passing, I consider the PR tested.

[eguzki]

LGTM

[tkan145]

Updated verification steps.

Because this has been approved. I will merge now