unknown process connecting on port 443... ipv6/networkmanager/latest git opensnitch.. systemd-resolved, firefox/firejail?

[andrewfader]

Running latest opensnitch-git on Arch 6.1.9, and as of the latest update I get a frequent unknown process connecting to ipv6 addresses on port 443... is this something to do with firefox? It seems to happen when I browse around in firefox (and firejail), how can I unmask this unknown process? If I deny it everything works fine but am I just breaking ipv6 loads for something? Using systemd-resolved

[gustavo-iniguez-goya]

Hi @andrewfader ,

Are you using "proc" or "ebpf" monitor method? (Preferences -> Nodes -> Process Monitor Method)

[andrewfader]

Yes it's the proc version, using a hardened kernel so it won't allow me to use ebpf

[gustavo-iniguez-goya]

Unfortunately using proc is expected to have sthis behaviour from time to time, specially if you're using firejail (or any other type of containerization).

What kernel are you using btw?

[andrewfader]

Linux 6.1.9-hardened1-2-hardened SMP PREEMPT_DYNAMIC Sun, 05 Feb 2023 06:19:26 +0000 x86_64 GNU/Linux

[gustavo-iniguez-goya]

Thank you @andrewfader , what's the name of the Manjaro package so I can install and test it?

[andrewfader]

It's https://aur.archlinux.org/packages/opensnitch-git from the aur

[andrewfader]

I've switched over to ebpf @gustavo-iniguez-goya but I still get an unknown process connecting, what should I do?

[andrewfader]

[andrewfader]

Hmm, actually, I can fix it by unchecking the "debug unknown connections" checkbox it seems.

[gustavo-iniguez-goya]

that seems to be a response to a SYN packet, see that the Source IP is a 192.168.x.x.

I know that we've this problem on some scenarios, but I've never known what causes it:

opensnitch/daemon/procmon/ebpf/find.go

Lines 25 to 33 in 74b6bc2

	if findAddressInLocalAddresses(dstIP) {
	// FIXME: systemd-resolved sometimes makes a TCP Fast Open connection to a DNS server (8.8.8.8 on my machine)
	// and we get a packet here with **source** (not detination!!!) IP 8.8.8.8
	// Maybe it's an in-kernel response with spoofed IP because resolved's TCP Fast Open packet, nor the response.
	// Another scenario when systemd-resolved or dnscrypt-proxy is used, is that every outbound connection has
	// the fields swapped:
	// 443:public-ip -> local-ip:local-port , like if it was a response (but it's not).
	// Swapping connection fields helps to identify the connection + pid + process, and continue working as usual
	// when systemd-resolved is being used. But we should understand why is this happenning.

One of the things I'll add to debug this and some other network scenarios is to log packet's bit fields and mark.

are you using VMs, containers, etc?

[andrewfader]

No VMs or containers, just a bunch of hardening. I changed this to use dhclient and dnsmasq instead of systemd-resolved. 192.168.50.77 is my own IP on the network.

[gustavo-iniguez-goya]

sorry, the destination port confused me. This doesn't seem to be the case I mentioned.

See if the ebpf hooks are added: # cat /sys/kernel/debug/tracing/kprobe_events

The output should be:

r16:kprobes/rtcp_v4_connect tcp_v4_connect
p:kprobes/ptcp_v6_connect tcp_v6_connect
r16:kprobes/rtcp_v6_connect tcp_v6_connect
p:kprobes/pudp_sendmsg udp_sendmsg
p:kprobes/pudpv6_sendmsg udpv6_sendmsg
p:kprobes/piptunnel_xmit iptunnel_xmit
p:kprobes/ptcp_v4_connect tcp_v4_connect

Also check the log file /var/log/opensnitchd.log for any errors loading the modules. There should be a log [eBPF] module loaded for opensnitch.o and opensnitch-procs.o

Anyway, if you could set LogLevel to DEBUG and post the log when you'll be prompted to allow these connections, it'd be useful to see what's going on that moment.

[andrewfader]

[2023-04-19 23:47:41]  DBG  new connection icmp6 => 0:2600:4041:524f:c600:2c3b:58f6:d98f:f47e -> 2600:4041:524f:c600:2c3b:58f6:d98f:f47e ():0 uid: 4294967295
[2023-04-19 23:47:41]  DBG  ebpf warning: [ebpf conn] unknown source IP: 2600:4041:524f:c600:2c3b:58f6:d98f:f47e
[2023-04-19 23:47:41]  DBG  [eBPF exit event] -> 25307
[2023-04-19 23:47:41]  DBG  new connection tcp => 58590:192.168.50.77 -> 46.232.210.30 ():17341 uid: 4294967295
[2023-04-19 23:47:41]  DBG  netlink socket error: Warning, no message nor error from netlink, or no connections found - 58590:192.168.50.77 -> 46.232.210.30:17341
[2023-04-19 23:47:41]  DBG  netlink socket error: Warning, no message nor error from netlink, or no connections found - 58590:192.168.50.77 -> 46.232.210.30:17341
[2023-04-19 23:47:41]  DBG  Searching for tcp6 netstat entry instead of tcp
[2023-04-19 23:47:41]  DBG  <== no inodes found for this connection: &netstat.Entry{Proto:"tcp", SrcIP:net.IP{0xc0, 0xa8, 0x32, 0x4d}, SrcPort:0xe4de, DstIP:net.IP{0x2e, 0xe8, 0xd2, 0x1e}, DstPort:0x43bd, UserId:-1, INode:-1}
[2023-04-19 23:47:41]  DBG  PID can't be read /proc/ -1 
[2023-04-19 23:47:41]  DBG  [-1] FindProcess() error: Unable to get process information
[2023-04-19 23:47:41]  DBG  Could not find process by its pid -1 for: 58590:192.168.50.77 (uid:-1) ->(tcp)-> 46.232.210.30:17341
[2023-04-19 23:47:41]  DBG  [eBPF exit event] -> 25308
[2023-04-19 23:47:41]  DBG  [ebpf] tcp map: 24 active items
[2023-04-19 23:47:41]  DBG  [ebpf] tcp6 map: 0 active items
[2023-04-19 23:47:41]  DBG  [ebpf] udp map: 362 active items
[2023-04-19 23:47:41]  DBG  [ebpf] udp6 map: 441 active items
[2023-04-19 23:47:42]  DBG  [eBPF exit event] -> 25309

[gustavo-iniguez-goya]

mmh, according to the logs , it looks like we're not finding the connection on the ebpf maps. It'd be interesting to dump the maps, but it's a tedious task. Anyway, I think we need a debug log to explicitly say that the connection hasn't been found via ebpf, because it's not clear to me.

Could you use tcpconnect or tcpconnect.bt (bpftrace package) to trace tcp connections and see if it outputs the name of the process that is creating these connections? To know what app (or even kmod) is initiating them and try to reproduce it.

There're ebpf exit events in the logs, so maybe there're also exec events prior to the connection.

[andrewfader]

@gustavo-iniguez-goya I believe the connections are being created by transmission-cli (bittorrent)

[gustavo-iniguez-goya]

by the way, are you still using a hardened kernel? do you usually suspend or hibernate your computer?

[andrewfader]

thanks for looking into it! I'm using the stock arch kernel now because the hardened kernel didn't support BPF. but I've recreated some of the hardened settings. and I do suspend at night so maybe that's related

[gustavo-iniguez-goya]

mmh, there's a bug after coming back from Suspend state:

for some reason (gobpf bug maybe) we're not able to delete intercepted connections from the eBPF maps, so when the maps reaches the maximum capacity (12000 elements right now), we cannot add new connections to the maps. As the new connections are not added, when a process opens an outbound connection, we don't find it via eBPF so we fallback to ProcFS, and in some scenarios like this one we're not able to obtain the process name.

I've added a script to dump the ebpf maps with the intercepted connections: https://github.com/evilsocket/opensnitch/blob/master/utils/scripts/debug-ebpf-maps.sh

Usage: bash ./debug-ebpf-maps.sh tcp (or tcpv6, udp, udpv6).

Dump your connections please, and see if the total elements of some map is >= 12000. If it's, then that's the reason of these "unknown" connections.

I've been suffering this bug since we added ebpf, and I've only found a temporal solution, which is restart opensnitch after coming back from suspend:

#!/bin/bash
# opensnitch - 2022-2023
#
# Due to a bug in gobpf, when coming back from suspend state, ebpf stops working.
# The temporal solution is to stop/start the daemon on suspend.
# 
# Copy it to /lib/systemd/system-sleep/ with any name and exec permissions.
#
if [ "$1" == "pre" ]; then
    service opensnitchd stop
elif [ "$1" == "post" ]; then
    service opensnitchd stop
    service opensnitchd start
fi

The downside of this work around is that you'll lose the temporary rules. But if it works I can add it to the repo.

There's an idea I haven't tested, which is to check for errors when adding a new connection to the maps. If the error is E2BIG (map full), reinitialize the map deleting all the elements. I also wrote a comment here:

[gustavo-iniguez-goya]

oops, I opened a thread describing the problem: #550
I didn't remember it O:)

[andrewfader]

Thank you @gustavo-iniguez-goya this is all very helpful. I'm pretty sure you have diagnosed the problem, but I will confirm later.