9 of integration tests fail (possibly with certificate errors)

[mgorny]

Describe the bug
I'm trying to package proxy.py for Gentoo, and I'm hitting tons of test failures. The best result I've gotten so far is via tox -e py311, which still gives 9 integration test failures:

FAILED tests/integration/test_integration.py::test_integration_with_interception_flags[--threaded --ca-cert-file /tmp/certificates/ca-cert.pem --ca-key-file /tmp/certificates/ca-key.pem --ca-signing-key /tmp/certificates/ca-signing-key.pem]
FAILED tests/integration/test_integration.py::test_integration_with_interception_flags[--threadless --local-executor 0 --ca-cert-file /tmp/certificates/ca-cert.pem --ca-key-file /tmp/certificates/ca-key.pem --ca-signing-key /tmp/certificates/ca-signing-key.pem]
FAILED tests/integration/test_integration.py::test_integration_with_interception_flags[--threadless --ca-cert-file /tmp/certificates/ca-cert.pem --ca-key-file /tmp/certificates/ca-key.pem --ca-signing-key /tmp/certificates/ca-signing-key.pem]
FAILED tests/integration/test_integration.py::test_modify_chunk_response_integration[--threaded --plugin proxy.plugin.ModifyChunkResponsePlugin --ca-cert-file /tmp/certificates/ca-cert-chunk.pem --ca-key-file /tmp/certificates/ca-key-chunk.pem --ca-signing-key /tmp/certificates/ca-signing-key-chunk.pem]
FAILED tests/integration/test_integration.py::test_modify_chunk_response_integration[--threadless --local-executor 0 --plugin proxy.plugin.ModifyChunkResponsePlugin --ca-cert-file /tmp/certificates/ca-cert-chunk.pem --ca-key-file /tmp/certificates/ca-key-chunk.pem --ca-signing-key /tmp/certificates/ca-signing-key-chunk.pem]
FAILED tests/integration/test_integration.py::test_modify_chunk_response_integration[--threadless --plugin proxy.plugin.ModifyChunkResponsePlugin --ca-cert-file /tmp/certificates/ca-cert-chunk.pem --ca-key-file /tmp/certificates/ca-key-chunk.pem --ca-signing-key /tmp/certificates/ca-signing-key-chunk.pem]
FAILED tests/integration/test_integration.py::test_modify_post_response_integration[--threaded --plugin proxy.plugin.ModifyPostDataPlugin --ca-cert-file /tmp/certificates/ca-cert-post.pem --ca-key-file /tmp/certificates/ca-key-post.pem --ca-signing-key /tmp/certificates/ca-signing-key-post.pem]
FAILED tests/integration/test_integration.py::test_modify_post_response_integration[--threadless --local-executor 0 --plugin proxy.plugin.ModifyPostDataPlugin --ca-cert-file /tmp/certificates/ca-cert-post.pem --ca-key-file /tmp/certificates/ca-key-post.pem --ca-signing-key /tmp/certificates/ca-signing-key-post.pem]
FAILED tests/integration/test_integration.py::test_modify_post_response_integration[--threadless --plugin proxy.plugin.ModifyPostDataPlugin --ca-cert-file /tmp/certificates/ca-cert-post.pem --ca-key-file /tmp/certificates/ca-key-post.pem --ca-signing-key /tmp/certificates/ca-signing-key-post.pem]

pytest log (190 KiB): test.txt

To Reproduce
tox -e py311

Expected behavior
Tests passing.

Version information

• OS: Gentoo Linux amd64
• proxy.py Version: a7077cf

Additional context
This part of the output looks suspicious:

* SSL certificate problem: invalid CA certificate
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection
curl: (60) SSL certificate problem: invalid CA certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
2024-06-22 07:49:41,136 - pid:912182 [W] server.wrap_client:833 - TLSV1_ALERT_UNKNOWN_CA: Client raised cert verification error for upstream: files.pythonhosted.org

[abhinavsingh]

@mgorny Thank you for the attached log file. Looking at it, here are the reason it fails:

1. sha256sum don't match --- Likely because the .whl file cannot be downloaded on your system

cat: downloaded.whl: No such file or directory
-: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
rm: cannot remove 'downloaded.whl': No such file or directory

2. SSL certificate problem: invalid CA certificate --- We pass -k flag to curl, but looks like it is still reporting an error about invalid CA certificate.

* SSL certificate problem: invalid CA certificate

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection
curl: (60) SSL certificate problem: invalid CA certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Could you check the curl setup itself once on your system. Does -k flag works for you. Also, can you try to manually download the .whl file used during integration tests. Unsure why integration test is unable to download this file. Does your system firewall or internet connection setup blocks access in any way?

[mgorny]

Could you check the curl setup itself once on your system. Does -k flag works for you.

Apparently it does, at least with a quick self-signed cert webserver:

$ curl https://192.168.1.1:18891/hello.txt
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.
$ curl -k https://192.168.1.1:18891/hello.txt
Hello world.

Also, can you try to manually download the .whl file used during integration tests.

$ curl -o /dev/null https://files.pythonhosted.org/packages/20/9a/e5d9ec41927401e41aea8af6d16e78b5e612bca4699d417f646a9610a076/Jinja2-3.0.3-py3-none-any.whl#sha256=077ce6014f7b40d03b47d1f1ca4b0fc8328a692bd284016f806ed0eaca390ad8
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  130k  100  130k    0     0   226k      0 --:--:-- --:--:-- --:--:--  228k

$ curl -o /dev/null https://files.pythonhosted.org/packages/88/78/e642316313b1cd6396e4b85471a316e003eff968f29773e95ea191ea1d08/proxy.py-2.4.0rc4-py3-none-any.whl#sha256=3d1921aab49d3464a712c1c1397b6babf8b461a9873268480aa8064da99441bc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  162k  100  162k    0     0   771k      0 --:--:-- --:--:-- --:--:--  788k

Unsure why integration test is unable to download this file. Does your system firewall or internet connection setup blocks access in any way?

No, or at least I'm not aware. Nor I recall having any network issues recently.

If that could help, today I'm using:

$ curl --version
curl 8.9.1 (x86_64-pc-linux-gnu) libcurl/8.9.1 OpenSSL/3.3.1 zlib/1.3.1 zstd/1.5.6 c-ares/1.32.2 libidn2/2.3.7 libpsl/0.21.5 nghttp2/1.62.1 nghttp3/1.4.0
Release-Date: 2024-07-31
Protocols: dict file ftp ftps http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smtp smtps tftp
Features: alt-svc AsynchDNS HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Largefile libz NTLM PSL SSL threadsafe TLS-SRP UnixSockets zstd

(I was using an older version when the report was filed)