🐛 BUG: IPv6 confusion (Listener is IPv4, but writing to IPv6 remote)

[erikschul]

What version of nebula are you using? (nebula -version)

1.9.4

What operating system are you using?

Linux

Describe the Bug

I have a node (A) that only supports ipv4, but the remote node (B) supports ipv6.

The nebula logs (A) says:

level=error msg="Failed to send handshake message" error="Listener is IPv4, but writing to IPv6 remote" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=727019211 localIndex=727019211 remoteIndex=0 udpAddr="<ipv6 address>:4242" vpnIp=...

I'm wondering if there may be multiple issues:

1)

I'm confused as to why (B) even has an ipv6 address, since the default value is 0.0.0.0 according to the docs.
Nebula shouldn't advertise addresses that it isn't listening on.

2)

Similarly, if listen on (A) doesn't support ipv6, it shouldn't try to connect to ipv6 addresses.

3)

I've tried specifying preferred_ranges: ['0.0.0.0/0'] on (A), but it still attempts to use ipv6.

Possibly related:

• 🐛 BUG: "Listener is IPv4, but writing to IPv6 remote" error is usually unhelpful #1227

Logs from affected hosts

Config files from affected hosts

[johnmaguire]

Hi @erikschul - thanks for the report. Can you share the configs from the affected hosts? Thanks.

[erikschul]

I've "solved" the problem by disabling ipv6 (net.ipv6.conf.all.disable_ipv6: 1), reasoning that if any node does not support ipv6, then all other nodes must support ipv4, and so little is lost by just using ipv6.
Nebula supports relay, but it's not clear whether ipv4 -> ipv6 is a supported use case.
Either way, ipv4 is fine for now I guess. But it does mean that the nodes won't be able to egress to ipv6 either. I guess that's why we're all still forced to using ipv4 :)

There are essentially no configs, everything is default values, except the lighthouse.

To reproduce, I suspect you will have 3 hosts:

1. lighthouse reachable by both (in my configs I only used ipv4 ips)
2. node with both ipv4 and ipv6, but listen.host defaults to 0.0.0.0, so ipv6 should be ignored?
3. node with only ipv4

[johnmaguire]

@erikschul Thanks for confirming. Sometimes the example config.yml's "defaults" do not match the code defaults, which is why I was hoping to see the config - however, I went and checked, and the defaults appear to match in both cases.

The behavior you're describing makes sense to me based on the way Nebula is currently coded. One way you could avoid this issue, without setting net.ipv6.conf.all.disable_ipv6 = 1, is to set local_allow_list and remote_allow_list to deny IPv6 addresses (or to allow IPv4 addresses.) The former will prevent advertising IPv6 addresses to the Lighthouse while the latter will prevent trying to form a tunnel to IPv6 addresses returned by the Lighthouse.

I'm going to close this as a dupe of #1227 and add some of the details you shared here. Thanks for filing this!

[erikschul]

I'm pretty sure that I did try to set both local_allow_list and remote_allow_list, and also restarted the lighthouse, and it still kept trying to connect to the ipv6 of the node.
I used 0.0.0.0/0.
I recommend checking the behavior.

But if the default listen.host is 0.0.0.0 then I argue that it would imply that ipv6 adresses should never be advertised?

I'm not sure that listen.host makes sense in combination with local_allow_list, so just always listen on all interfaces that match local_allow_list, and remove listen.host.

Either way, these are just suggestions, I'm happy to just use ipv4 for now.

Thanks for everything!

[johnmaguire]

@erikschul listen.host and local_allow_list have some similarities but serve different goals. listen.host indeed specifies which address to bind to / listen on. So while 0.0.0.0 will listen on all IPv4 interfaces and :: will listen on both IPv4 and IPv6, you could specify a specific address, such as 127.0.0.1 (or perhaps an Elastic IP.)

Today, Nebula doesn't try to make any decisions based on the listen.host value in terms of which addresses it reports to the Lighthouse. All known addresses are reported, and the Lighthouse learns the public IP through the source address in the IP packet.

local_allow_list is used to further restrict which addresses are sent to the Lighthouse. This is primarily useful for things like excluding Docker interfaces or other VPN interfaces that we don't want nodes to try to reach out to us over.

But if the default listen.host is 0.0.0.0 then I argue that it would imply that ipv6 adresses should never be advertised?

Indeed, that seems like a reasonable improvement, and it's captured in ticket #1227.

I'm pretty sure that I did try to set both local_allow_list and remote_allow_list, and also restarted the lighthouse, and it still kept trying to connect to the ipv6 of the node.

Without seeing the configs you used, it's hard to say why that didn't work. But it's worth noting that while the Lighthouse has one cache, which would've been cleared by restarting, so does each node. So you'd really want to restart all three nodes to ensure the change to local_allow_list takes effect.

Hope that helps!