Help with listener program

[shulltronics]

Hi! Thanks for making this awesome tool. I'm using it for a university project, exploring the security of phone app communications. I've been reading though all of the documentation and running a lot of experiments, but am getting stuck with my listener program, which for now I just want to be a simple packet pass-through.

For my setup, I have a laptop forming a hotspot on one network interface, and connected to the internet via another. I am doing internet sharing by configuring some iptables rules (see
iptables-setup.txt if interested) I have SSLproxy sitting between, processing the packets.

I have been able to get sslsplit to work, as well as SSLproxy in passthrough mode, but cannot seem to get my listener program to properly communicate with SSLproxy.

I'm starting SSLproxy with sslproxy -l connections.log -j tmp/sslproxy/ -S sslproxy-logs/ -X sslproxy.pcap -k ca.pem -c ca.crt -f sslproxy.conf, and my config file is as follows:

Daemon no
Debug yes
DebugLevel 4
LogStats yes
StatsPeriod 1
ConnIdleTimeout 120
VerifyPeer no

ProxySpec {
    Proto ssl
    Addr 0.0.0.0
    Port 8443
    Divert yes
    DivertAddr 127.0.0.1
    DivertPort 10101
}

ProxySpec {
    Proto tcp
    Addr 0.0.0.0
    Port 8080
    #Passthrough yes
    #Divert no
    Divert yes
    DivertAddr 127.0.0.1
    DivertPort 10101
}

My listener script is as follows:

import asyncio
import socket
import re

LISTENER_ADDR = '127.0.0.1'
LISTENER_PORT = 10101
num_connections = 0

def parse_sslproxy_line(payload):
    """
    Extract the SSLproxy line from a payload string
    field denoted which (addr, port) pair to return
    """
    # First check that this packet contains the SSLproxy line
    if payload[0:10] != "SSLproxy: ":
        return None
    string = payload[10:]       # remove the line
    # get the comma delimited fields
    parts = re.split(',', string)
    # these regexs match the ip addresses and ports
    addr_regex = r"\[(.*?)\]"
    port_regex = r":([0-9]*)"
    sslproxy_addr = re.search(addr_regex, parts[0]).group(1)
    sslproxy_port = re.search(port_regex, parts[0]).group(1)
    src_addr      = re.search(addr_regex, parts[1]).group(1)
    src_port      = re.search(port_regex, parts[1]).group(1)
    dst_addr      = re.search(addr_regex, parts[2]).group(1)
    dst_port      = re.search(port_regex, parts[2]).group(1)
    encryption    = parts[3]
    return (sslproxy_addr, int(sslproxy_port), src_addr, int(src_port), dst_addr, int(dst_port), encryption)

async def forward_packets(reader, writer):
    """
    This coroutine is called when a connection is established to this program
    The data is read from the stream, the SSLproxy line is parsed (if present)
    and then the data is sent back to SSLproxy on the appropriate address:port
    """
    # Co-routine specific variables
    global num_connections
    sslproxy_ip = None
    sslproxy_port = None
    while True:
        data = await reader.readline()
        if not data:
            break
        try:
            message = data.decode()
            sslp_addrs = parse_sslproxy_line(message)
            if sslp_addrs:
                (sslp_ip, sslp_port, _, _, _, _, _) = sslp_addrs
                print("  found SSLproxy line => {0}:{1}".format(sslp_ip, sslp_port))
                print("  number of connections: {}".format(num_connections))
                sslproxy_ip = sslp_ip
                sslproxy_port = sslp_port
                num_connections += 1
        except Exception as e:
            print("Error decoding packet as utf-8: ", str(e))

        try:
            w = socket.create_connection((sslproxy_ip, sslproxy_port))
            w.send(data)
            w.close()
        except Exception as e:
            print("Couldn't connect back to SSLproxy: ", str(e))

async def main():
    server = await asyncio.start_server(
        forward_packets, LISTENER_ADDR, LISTENER_PORT)

    addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets)
    print(f'Serving on {addrs}')

    async with server:
        await server.serve_forever()

asyncio.run(main())

After starting my program and SSLproxy, and connecting my phone to generate some traffic, the program outputs are as follows:
listener-output.txt
sslproxy-output.txt

I know this is a lot to read, but I would really appreciate any guidance you could give me. I have done a lot to try to get this working and I think I just need a little help. Thanks so much!!

[sonertari]

Can you read this comment of mine to another user on another thread, and try again? I read in your code that you parse the SSLproxy line, but I think you close the connection to sslproxy. Your listening program should keep sslproxy connections (on both sides) open until sslproxy decides to close them. Also, please see the sample lp in the sources.

[shulltronics]

@sonertari: Thank you for your help. After looking more closely at that issue, I was able to get a "kind of" working example. In this code, I asynchronously listen for new connections from SSLproxy, parse the SSLproxy line, and then start forwarding the data between the two SSLproxy ports (client -> server, and vice versa).

If the EOF character is received, I return from the coroutines, essentially ending the connection. (I'm not sure if this is the proper action to take or not).

This works somewhat, in the sense that I can load most webpages through my listener script; however, the loading bar doesn't always complete, and some applications don't work at all. The main issue my experiments have shown is that sometimes the SSLproxy line doesn't seem to be present in the first line from SSLproxy. Furthermore, it seems that sometimes the connections that come from SSLproxy immediately return the EOF character.

I have explored the lp example, and see where the SSLproxy line is parsed in prototcp.c. I believe I am attempting to parse the line properly, and can't understand why some of the connections coming from SSLproxy seem to close right away of not contain the SSLproxy line.

So I guess my questions now are:

• Are there cases when the SSLproxy line would come later in a packet stream than the first \n terminated line?
• Are there cases when SSLproxy would open a connection to my listener program but then immediately close it?

Thanks again for your help, and here is my code in case it's helpful:

import asyncio
import socket
import re

# Address to listen on
LISTENER_ADDR = '127.0.0.1'
LISTENER_PORT = 10101
# Global state keeping variables TODO make this the PacketMaster class, keeping much more detailed track of things
num_connections = 0
sslproxy_ports = []

def parse_sslproxy_line(payload):
    """
    Extract the SSLproxy line from a payload string
    """
    # First check that this packet contains the SSLproxy line
    try:
        sslproxy_line_index = payload.index("SSLproxy: ")
    except:
        return None
    string = payload[sslproxy_line_index+10:] # remove the identifier
    # get the comma delimited fields
    parts = re.split(',', string)
    # these regexs match the ip addresses and ports
    addr_regex = r"\[(.*?)\]"
    port_regex = r":([0-9]*)"
    sslproxy_addr = re.search(addr_regex, parts[0]).group(1)
    sslproxy_port = re.search(port_regex, parts[0]).group(1)
    src_addr      = re.search(addr_regex, parts[1]).group(1)
    src_port      = re.search(port_regex, parts[1]).group(1)
    dst_addr      = re.search(addr_regex, parts[2]).group(1)
    dst_port      = re.search(port_regex, parts[2]).group(1)
    # single letter encryption scheme
    encryption    = parts[3][0]
    return (sslproxy_addr, int(sslproxy_port), src_addr, int(src_port), dst_addr, int(dst_port), encryption)

async def server_client_forward(to_client, from_server):
    """
    This coroutine is called each time an SSLproxy line is received and parsed by `client_server_forward`
    It is responsible for asyncronously getting server responses from SSLproxy, and forwarding them back
    for the client to receive
    """
    (conn_ip, conn_port) = to_client.get_extra_info('peername')
    print(f"  {conn_port}: started server_client_forward task")
    while not from_server.at_eof():
        # Get server->client data from SSLproxy, and send back
        server_client_data = await from_server.readline()
        to_client.write(server_client_data)
        await to_client.drain()
    print(f"  {conn_port}: exiting server_client_forward!")

async def client_server_forward(from_client, _to_client):
    """
    This coroutine is called when SSLproxy establishes a connection with this program
    * The data going from client->server is read from `from_client`,
      the first packet of that data should contain the SSLproxy line.
    * After parsing the SSLproxy line, another bidirectional socket stream is established
      with SSLproxy on the assigned port, assigned to `to_server` and `from_server`
    * The client->server data is the written to `to_server`.
    * The server->client response is then received from SSLproxy via `from_server`,
      and forwarded on to `to_client`.
    """
    # Get some identifying info about the connection
    global num_connections, sslproxy_ports
    (conn_ip, conn_port) = _to_client.get_extra_info('peername') # TODO would it be equivalent to use from_client?
    print(f"Connection from SSLproxy port {conn_port}")
    sslproxy_ports.append(conn_port)
    num_connections += 1
    print(f"  {conn_port}: number of connections: {num_connections}")
    # First, get the SSLproxy line and populate the upstream address
    sslproxy_ip = None
    sslproxy_port = None
    data = await from_client.readline()
    try:
        message = data.decode()
    except Exception as e:
        print("  ERROR decoding payload (returning):", str(e))
        return
    sslp_addrs = parse_sslproxy_line(message)
    if sslp_addrs:
        (sslp_ip, sslp_port, src_ip, src_port, dst_ip, dst_port, ssl) = sslp_addrs
        print(f"  {conn_port}: found SSLproxy line => {sslp_ip}:{sslp_port}")
        print(f"  {conn_port}: src address => {src_ip}:{src_port}")
        print(f"  {conn_port}: dst address => {dst_ip}:{dst_port}")
        print(f"  {conn_port}: original encryption: {ssl}")
        sslproxy_ip = sslp_ip
        sslproxy_port = sslp_port
    ### At this point, we have the port of where to send/receive data to/from the server,
    #   but we need this to also run asyncronously.
    # Create an async connection to SSLproxy as specified in the SSLproxy line
    try:
        print(f"  {conn_port}: connecting to SSLproxy at {sslproxy_ip}:{sslproxy_port}")
        # TODO sometimes I have to sway the following two lines for the connections to work??
        (from_server, to_server) = await asyncio.open_connection(sslproxy_ip, sslproxy_port)
        #(from_server, to_server) = await asyncio.open_connection(sslproxy_ip, sslproxy_port, local_addr=(sslproxy_ip, sslproxy_port))
        # Schedule the server_client_forward task, passing the relevant connections over
        loop = asyncio.get_running_loop()
        loop.create_task(server_client_forward(_to_client, from_server)) # the ONLY spot we use _to_client in this coro
    except Exception as e:
        print(f"  {conn_port}: ERROR: couldn't connect back to SSLproxy:", str(e))
        return
    # Send this packet back to SSLproxy
    #print("  {0}: [pre-loop] writing {1} bytes to to_server.".format(conn_port, len(data)))
    to_server.write(data)
    await to_server.drain()
    # Now forward all subsequent packets to the open connection, in both directions
    while not from_client.at_eof():
        # Get client->server data from SSLproxy, and send back
        client_server_data = await from_client.readline()
        to_server.write(client_server_data)
        await to_server.drain()
    print(f"  {conn_port}: task_a at EOF, exiting loop!")


async def main():
    server = await asyncio.start_server(
        client_server_forward, LISTENER_ADDR, LISTENER_PORT)
    addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets)
    print(f'Serving on {addrs}')
    async with server:
        await server.serve_forever()

asyncio.run(main())

[shulltronics]

Upon further investigation, I found something interesting. The connections that are opened to my listener script that do not contain the SSLproxy line seem to correspond to BEV_EVENT_ERROR lines in the debug logs of SSLproxy.

I have read a couple of other issues that mention this error, but they didn't seem super relevant to me until now. I am using certificates on my proxy machine that I generated with openssl, and copied/installed the certificate on my client machine. Seeing as some of the ssl connections succeed, it doesn't seem like an issue with the ssl certificates. Something that I had read earlier made me set the VerifyPeer option in the config to no, but I can't seem to find again where I discovered that.

I've attached the debug log from SSLproxy (report.txt), and the logs from my two listener scripts (identical scripts, just listening on different ports, one for tcp protocol, and the other for ssl protocol). From what I can tell, the times when the listener script cannot parse an SSLproxy line seem to correspong to the BEV_EVENT_ERRORs in the SSLproxy debug.

Can you see any reason for the errors? Let me know of any other experiments/details you need me to provide. Thanks again (in advance) for helping me out.

report.txt
ssl-logs.txt
tcp-logs.txt

[sonertari]

As mentioned in README, "SSLproxy inserts in the first packet the address and port it is expecting to receive the packets back from the program." So, sslproxy does not insert any further SSLproxy lines in any other packets. So, your program should maintain the state of each connection, and use the info in the SSLproxy line in the very first packet. I think the issues you observe are related with that.

[shulltronics]

Thanks for getting back with me. I'm not saying that my listener program is flawless, but I think the issues I'm having are more related to the BEV_EVENT_ERRORs I'm seeing. I'm putting the link to this file here just for reference. I will follow up if I discover anything that helps me solve this.

sonertari/UTMFW#4

[shulltronics]

I've migrated my project over to using mitmproxy, which I think is a better solution for me right now. Thanks again, for your great work and willingness to help me!