Working with servers with self-signed sertificates.

[ghost]

Sorry, it is a long story, but I tried to mention all the most important points here.

I have a server ("target") which works via https but has self-signed certificates (and just in case this certificate is outdated). In general, I do not have the access to root-certificate which was used to build "target"-server certificate.

And I am trying to use proxy.py as something like proxy authentication - I need to add some headers (like, Authorization) in order to authenticte on the "target"-server.

I am running proxy.py so:

proxy \
                --plugins \
                test_plugin.ModifyPlugin \
                --port 8899 \
                --log-level d \
                --ca-key-file ca-key.pem \
                --ca-cert-file ca-cert.pem \
                --ca-signing-key ca-signing-key.pem

ca-key.pem, ca-cert.pem, ca-signing-key.pem were made via make ca-certificates.

Now I am trying to perform a request to "target"-server:

curl -v -x localhost:8899 https://<server_ip_here>

*localhost:8899 - is the address of proxy.py instance.

I get something like this:

$ curl -v -x localhost:8899 https://<server_ip_here>
*   Trying 127.0.0.1:8899...
* Connected to (nil) (127.0.0.1) port 8899 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to <server_ip_here>:443
> CONNECT <server_ip_here>:443 HTTP/1.1
> Host: <server_ip_here>:443
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
* error:0A000126:SSL routines::unexpected eof while reading
* Closing connection 0
curl: (35) error:0A000126:SSL routines::unexpected eof while reading

I have same error when ask curl to ignore SSL checking (-k option):

$ curl -k -v -x localhost:8899 https://<server_ip_here>/login
*   Trying 127.0.0.1:8899...
* Connected to (nil) (127.0.0.1) port 8899 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to <server_ip_here>:443
> CONNECT <server_ip_here>:443 HTTP/1.1
> Host: <server_ip_here>:443
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
* error:0A000126:SSL routines::unexpected eof while reading
* Closing connection 0
curl: (35) error:0A000126:SSL routines::unexpected eof while reading

In the proxy.py log I have the following warning:

2024-02-07 10:42:55,982 - pid:2621861 [D] server.connect_upstream:567 - Connecting to upstream <server_ip_here>:443
2024-02-07 10:42:55,984 - pid:2621861 [D] server.connect_upstream:602 - Connected to upstream <server_ip_here>:443
2024-02-07 10:42:58,169 - pid:2621861 [W] server.wrap_server:765 - ssl.SSLCertVerificationError: Server raised cert verification error for upstream: <server_ip_here>

The error is generated here

proxy.py/proxy/core/connection/server.py

Line 62 in 30574fd

self._conn = ctx.wrap_socket(

and it looks quite correct for this case - sercificate is "bad" and cannot be used to establish a secure connection.

But I need an option to work even with such servers. I have tried to make some workarounds for this case, more or less it works but it looks clunky and ugly and I am asking community-help besause I think there is a proper and better way to do this.

Right now I redefined TcpServerConnection.The main change is ctx.verify_mode = ssl.VerifyMode.CERT_NONE:

class TcpServerConnectionWithSelfSigned(TcpServerConnection):

    def wrap(
            self,
            hostname: Optional[str] = None,
            ca_file: Optional[str] = None,
            as_non_blocking: bool = False,
            # Ref https://github.com/PyCQA/pylint/issues/3691
            verify_mode: ssl.VerifyMode = ssl.VerifyMode.CERT_REQUIRED,     # pylint: disable=E1101
    ) -> None:
        return self._wrap_self_signed(hostname, as_non_blocking)

    def _wrap_self_signed(self, hostname, as_non_blocking):
        ctx = ssl.SSLContext()
        ctx.options |= ssl.OP_NO_SSLv2 | ssl.OP_NO_SSLv3 | ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1
        ctx.verify_mode = ssl.VerifyMode.CERT_NONE
        self.connection.setblocking(True)
        self._conn = ctx.wrap_socket(
            self.connection,
            server_hostname=hostname,
        )
        if as_non_blocking:
            self.connection.setblocking(False)

In order to enable TcpServerConnectionWithSelfSigned I had to copy-paste more code from HttpProxyPlugin:

class HttpProxyPluginOverridden(HttpProxyPlugin):

    def connect_upstream(self) -> None:
         # ... content skipped ...
                if self.flags.enable_conn_pool:
                    assert self.upstream_conn_pool
                    with self.lock:
                        created, self.upstream = self.upstream_conn_pool.acquire(
                            (text_(host), port),
                        )
                else:
                    created, self.upstream = True, TcpServerConnectionWithSelfSigned(
                        text_(host), port,
                    )
         # ... content skipped ...

After that I need to run proxy.py like:

proxy \
                --plugins \
                test_plugin.HttpProxyPluginOverridden 
                test_plugin.ModifyPlugin \
                --disable-http-proxy \
                --port 8899 \
                --log-level d \
                --ca-key-file ca-key.pem \
                --ca-cert-file ca-cert.pem \
                --ca-signing-key ca-signing-key.pem

This "update" allows me to avoid exception in wrap_socket-function. But after that when I am making a new request I have another exception here:

2024-02-07 11:35:22,429 - pid:2623584 [E] base_events.default_exception_handler:1771 - Task exception was never retrieved
future: <Task finished name='Task-1' coro=<Threadless._run_forever() done, defined at /home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/core/work/threadless.py:376> exception=KeyError('subject')>
Traceback (most recent call last):
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/core/work/threadless.py", line 380, in _run_forever
    if await self._run_once():
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/core/work/threadless.py", line 364, in _run_once
    teardown = task.result()
               ^^^^^^^^^^^^^
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/http/handler.py", line 149, in handle_events
    teardown = await self.handle_readables(readables)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/http/handler.py", line 219, in handle_readables
    teardown = await super().handle_readables(readables)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/core/base/tcp_server.py", line 210, in handle_readables
    r = self.handle_data(data)
        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/http/handler.py", line 172, in handle_data
    if self._parse_first_request(data):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/http/handler.py", line 300, in _parse_first_request
    output = self.plugin.on_request_complete()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/http/proxy/server.py", line 513, in on_request_complete
    return self.intercept()
           ^^^^^^^^^^^^^^^^
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/http/proxy/server.py", line 742, in intercept
    teardown = self.wrap_client()
               ^^^^^^^^^^^^^^^^^^
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/http/proxy/server.py", line 797, in wrap_client
    generated_cert = self.generate_upstream_certificate(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/http/proxy/server.py", line 731, in generate_upstream_certificate
    self.gen_ca_signed_certificate(cert_file_path, certificate)
  File "/home/kav/AK/proxy_py/.venv/lib/python3.11/site-packages/proxy/http/proxy/server.py", line 641, in gen_ca_signed_certificate
    upstream_subject = {s[0][0]: s[0][1] for s in certificate['subject']}
                                                  ~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'subject'

If I understand correctly this happens when proxy.py tries to create a new certificate based on data of original "target"-server certificate and gen_ca_signed_certificate expects a "target"-certificate data in dictionary form but gets empty dictionary. And It gets empty dictionary because getpeercert returns empty dict for non-validated certificates (https://docs.python.org/3/library/ssl.html#ssl.SSLSocket.getpeercert):

If the certificate was not validated, the dict is empty

I have tried to workaround this somehow too and stopped on the following (but very very clunky) way - I redefined wrap_client too.
The main idea is to pass binary_form=True to getpeercert. This allows me to get certificate data (in DER-format) even it is not validated (proof: https://github.com/python/cpython/blob/3.11/Modules/_ssl.c#L1841), then I converts certificate data from DER to PEM via openssl command, parses certificate data to dictionary via "undocumented" test_decode_cert-function (because I cannot call _decode_certificate directly, https://github.com/python/cpython/blob/3.11/Modules/_ssl.c#L1769) and finally passes obtained dictionary to generate_upstream_certificate. The example of code is bellow:

class HttpProxyPluginOverridden(HttpProxyPlugin):

    def _der_2_pem(self, der_path, pem_path):
        # openssl x509 -inform der -in filename -out filename.pem
        command = ['openssl', 'x509', '-inform', 'der', '-in', der_path, '-out', pem_path]
        return run_openssl_command(command, 10000)

    def _obtain_certificate(self) -> dict:
        import _ssl
        with (
            NamedTemporaryFile(suffix='.der') as der_tf,
            NamedTemporaryFile(suffix='.pem') as pem_tf
        ):
            binary_cert = self.upstream.connection.getpeercert(binary_form=True)
            der_tf.write(binary_cert)
            der_tf.flush()

            self._der_2_pem(der_tf.name, pem_tf.name)
            return _ssl._test_decode_cert(str(pem_tf.name))

    def wrap_client(self) -> bool:
        assert self.upstream is not None and self.flags.ca_signing_key_file is not None
        assert isinstance(self.upstream.connection, ssl.SSLSocket)
        do_close = False
        try:
            # TODO: Perform async certificate generation
            cert_as_dict = self._obtain_certificate()
            generated_cert = self.generate_upstream_certificate(
                cast(Dict[str, Any], cert_as_dict),
            )
            self.client.wrap(
                self.flags.ca_signing_key_file, generated_cert, ca_cert=self.flags.ca_cert_file
            )

After that I am able to proxy requests to servers with self-signed (and not valid) certificates but as I said it looks ugly and I think there is a better way to do this. I will be very apreciated to any suggestions (and critica, like for security holes).