✨ Load Balancer IPv6 as apiserver endpoint

[JochemTSR]

What this PR does / why we need it:
A cluster can be provisioned with a HCloud Load Balancer, which is a convenient way to present a singular entrypoint to the API server. Currently, The API server endpoint used by e.g. the kubelet is always set to the Load Balancer's IPv4 address (if no explicit hostname was configured). This means that nodes need to have a public-facing IPv4 interface and address, as crucial Kubernetes components can otherwise not reach the API server and will fail to be properly provisioned.

Meanwhile, nodes without an IPv4 is already possible with CAPH HCloudMachineTemplates by setting spec.template.spec.publicNetwork.enableIPv4 to false. Doing so can yield worthwile cost savings if the provisioned nodes are rather small (e.g. CAX11), as one is charged a flat fee for using an IPv4 address. But, as mentioned before, such nodes would fail to provision if a Load Balancer is used. This PR introduces a feature that allows the API server endpoint to be set to the Load Balancer's public IPv6 instead of the IPv4. Doing so allows for successful initialization of a cluster without public IPv4 addresses on the nodes. Manifests to create a working example cluster can be found at (https://github.com/JochemTSR/clusterapi-playground/tree/main/ipv6dev).

It should be noted that while Cluster API does exactly what is asked I had to pull an nat64 trick to make GitHub (and therefore kubeadm initialization) work with IPv6-only nodes, as GH apparently doesn't do IPv6 yet. Until I find a more elegant way to ensure provisioning succeeds with IPv6 only, I'll hold off on proposing a flavor for this.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1218

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squash commits
• include documentation
• add unit tests

[janiskemper]

thanks a lot @JochemTSR for your contribution! Is this ready to be tested and reviewed? If so, I would ask @guettli to have a look!

[JochemTSR]

@guettli all seems good on my end and tests are passing, could you have a look?

[janiskemper]

@JochemTSR sorry for letting you wait for so long. I'll do it myself now:

1. Can you add a template for this feature to https://github.com/syself/cluster-api-provider-hetzner/tree/main/templates/cluster-templates?
2. Can you maybe even add an e2e test for this (which is based on these templates) in https://github.com/syself/cluster-api-provider-hetzner/tree/main/test/e2e? You should probably also switch off the IPv4 addresses of the HCloudMachines to ensure that everything works only in IPv6.

Does this make sense to you? Do you think you can do also the e2e testing part?

[guettli]

@JochemTSR my teammate Janis asked you some question. Are still interested in getting your PR merged?

Please let us know your reasoning (even if you don't care anymore).

[rbjorklin]

For what it's worth I'd be interested in seeing this merged. I could probably help whip up the templates requested if that would help move things along.

[janiskemper]

are you also able to write Go code? Then we could add an e2e test for it as well. That would be amazing! If not, any contribution is helpful :)

[JochemTSR]

Hi all, I aploogize for going AWOL for so long. It's been busy, I kind of forgot about this project and didn't have email notifications on. I made sure to enable them now.

I'd be glad to work with @rbjorklin to create some templates and working e2e tests. However I would like some input on the way I worked around GitHub's lack of IPv6 support. If a node cannot reach GitHub, the kubeadm initialization will not complete. I made the following modification to the cluster template to get it to work (you can also see for yourself in the linked repo):

    spec:
      files:
      - content: | # CHANGED: github does not do ipv6(???), so use a nat64 solution
          [Resolve]
          DNS=2a00:1098:2b::1 2a01:4f8:c2c:123f::1 2a00:1098:2c::1
          Domains=~github.com githubusercontent.com
        owner: root:root
        path: /etc/systemd/resolved.conf.d/nat64.conf
        permissions: "0644"

What this basically does is send github-related traffic to a service that translates ipv6 traffic to ipv4 traffic and back. In this case, traffic is sent to a service hosted by https://nat64.net/ .

In my honest opinion, using a public nat64 solution might not be quite production-worthy, and if you agree we might have to table this IPv6 feature until Github is routable through IPv6 or until we can come up with a better solution. I would greatly appreciate your input on this matter @janiskemper @guettli . (or perhaps @rbjorklin got it to work some other way and we can ignore this alltogether?)

Regards,

[guettli]

Please use require.Equal(t, a, b)

[guettli]

Hi all, I aploogize for going AWOL for so long. It's been busy, I kind of forgot about this project and didn't have email notifications on. I made sure to enable them now.

I'd be glad to work with @rbjorklin to create some templates and working e2e tests. However I would like some input on the way I worked around GitHub's lack of IPv6 support. If a node cannot reach GitHub, the kubeadm initialization will not complete. I made the following modification to the cluster template to get it to work (you can also see for yourself in the linked repo):

    spec:
      files:
      - content: | # CHANGED: github does not do ipv6(???), so use a nat64 solution
          [Resolve]
          DNS=2a00:1098:2b::1 2a01:4f8:c2c:123f::1 2a00:1098:2c::1
          Domains=~github.com githubusercontent.com
        owner: root:root
        path: /etc/systemd/resolved.conf.d/nat64.conf
        permissions: "0644"

What this basically does is send github-related traffic to a service that translates ipv6 traffic to ipv4 traffic and back. In this case, traffic is sent to a service hosted by https://nat64.net/ .

In my honest opinion, using a public nat64 solution might not be quite production-worthy, and if you agree we might have to table this IPv6 feature until Github is routable through IPv6 or until we can come up with a better solution. I would greatly appreciate your input on this matter @janiskemper @guettli . (or perhaps @rbjorklin got it to work some other way and we can ignore this alltogether?)

@JochemTSR thank you very much for nat64.conf. I have not tested it, but it looks like a good solution.

Up to now I have only seen a unit-test in the PR. BTW, could you please use require.Equal(), this makes it easier to read.

@JochemTSR Please elaborate your overall goal. Why do you want ipv6 support? I know this sounds like a strange question, but I would like to understand why you want that feature.

What are your plans for the kubernetes-network? Do you want to use ipv4-only, dual, or ipv6-only? Please elaborate why you want it.

At the moment our customers don't ask for that feature, so we work on other parts. But if there are good reasons for that, then we might put this on our roadmap.

[JochemTSR]

@guettli Thank you for your response. I should have some time this weekend to incorporate the modified configs into a proper template.

The motivation behind this feature was discussed in the linked issue (#1218). I believe this might also fix closed issues #899 and #632 .