Also add labels to Hetzner Server from NodeTemplate #109

NotANormalNerd · 2021-01-09T17:57:18Z

Add the Labels defined by the Node Templates to Hetzner servers, which add the possibilities to use label selectors for load Balancers as requested in #100

mxschmitt

could it be, that you forgot to commit/push the changes in the template? Since the setLabels function needs to be called somewhere and I think that was from the template. But maybe I'm also wrong.

NotANormalNerd · 2021-01-10T14:21:56Z

The setLabels gets called by the form-user-labels . here

ui-driver-hetzner/component/template.hbs

Line 100 in 1d338fc

    
           {{form-user-labels initialLabels=labelResource.labels setLabels=(action 'setLabels') expandAll=expandAll expand=(action expandFn) }}

We also have to call super on the NodeDriver to set the labels in rancher/k8s as well.

I had to basically set rancher to TRACE since there is no documentation on the custom-node-drivers whatsoever. Took me way to long to find all source code and figure stuff out.

mxschmitt · 2021-01-10T22:30:55Z

The setLabels gets called by the form-user-labels . here

ui-driver-hetzner/component/template.hbs

Line 100 in 1d338fc

{{form-user-labels initialLabels=labelResource.labels setLabels=(action 'setLabels') expandAll=expandAll expand=(action expandFn) }}

We also have to call super on the NodeDriver to set the labels in rancher/k8s as well.
I had to basically set rancher to TRACE since there is no documentation on the custom-node-drivers whatsoever. Took me way to long to find all source code and figure stuff out.

thanks for the explanation, merged!

KochMario · 2021-01-12T21:30:50Z

Hi there, I just tried including the server label option in my setup as well.
It appears however, that this option is not mirrored yet in the UI itself? Im using the docker-machine-driver-hetzner with Rancher and the following is my current Node Driver Config.

So I'm using the latest (3.2) version, which includes --hetzner-server-label flag, the UI itself doesn't seem to provide that option though.
When trying to instantiate a new node with a template I receive the following error

According to your code @mxschmitt , it seems that you're setting all provided labels as serverLabels. Or am I mistaken about that part?

setLabels: function setLabels(labels) {
    var labels_list = labels.map(function (l) {
      return l.key + "=" + l.value;
    });
    this.set('model.hetznerConfig.serverLabel', labels_list);

    this._super(labels);
  }

I tried setting a label via the labels section in the Node Template GUI.
Am I missing something?

NotANormalNerd · 2021-01-13T08:57:41Z

No you are right, we use all k8s labels we set for the nodes and set them on the virtual servers at Hetzner. It seems that the set Labels function is called even without sepcifying any labels.

In my tests there always was one so I haven't tested it without any labels, my bad.

In the meantime, you can just specify any label and the setup should work fine. @KochMario
Would you also mind showing us the node template you setup?

KochMario · 2021-01-14T13:27:51Z

I think I figured out the issue.
I tried adding the newly created Node Template (with Hetzner Server Labels) to an already existing Cluster (which was setup using an old hetzner node driver).
This supposedly wasn't compatible. The cluster is actually broken now and I can't even create any new nodes with the old Node Templates.

Creating an entirely new cluster from scratch with the newly added Node Templates does work though. The provided labels are being correctly applied to the created Hetzner Servers.

It's definitely kind of a bummer that the version upgrade effectively corrupted the cluster, but since we're still in trial & setup mode it's not that big of a deal for us.
Hetzner Server Labels seem to work, so I'm happy :)
Thanks for your work in enabling that! @NotANormalNerd

mxschmitt · 2021-01-14T13:32:07Z

is it possible to make it not a breaking change? maybe pass empty ones by default @NotANormalNerd

NotANormalNerd · 2021-01-14T16:05:40Z

I am really sorry for that. Maybe we can have versioned UIs? So that we don't break existing installations?
Yes. We can do something. My javascript-fu is not very low. So we could just check if the labels_list is empty, and if it is we can ommit and delete the this.set('model.hetznerConfig.serverLabel', labels_list); @mxschmitt

@KochMario Again I am sorry, should have checked that. But the old templates should work without problems. The driver is called with the HetznerConfig that is saved in Rancher, which should not have serverLabels saved. If you saved the old ones without labels this should at least have the rancher labels creator=norman saved. If you removed the default labels, then none are saved. You can get around that by just adding a label.

ssimeth · 2021-02-26T15:45:01Z

I think I figured out the issue.
I tried adding the newly created Node Template (with Hetzner Server Labels) to an already existing Cluster (which was setup using an old hetzner node driver).
This supposedly wasn't compatible. The cluster is actually broken now and I can't even create any new nodes with the old Node Templates.

Creating an entirely new cluster from scratch with the newly added Node Templates does work though. The provided labels are being correctly applied to the created Hetzner Servers.

It's definitely kind of a bummer that the version upgrade effectively corrupted the cluster, but since we're still in trial & setup mode it's not that big of a deal for us.
Hetzner Server Labels seem to work, so I'm happy :)
Thanks for your work in enabling that! @NotANormalNerd

I have the same problem, but in a production setup. Do I have any chance to solve the problem? It appears that the existing cluster is using the old driver and only new clusters use the upgraded driver.

KochMario · 2021-02-28T09:23:52Z

I think I figured out the issue.
I tried adding the newly created Node Template (with Hetzner Server Labels) to an already existing Cluster (which was setup using an old hetzner node driver).
This supposedly wasn't compatible. The cluster is actually broken now and I can't even create any new nodes with the old Node Templates.
Creating an entirely new cluster from scratch with the newly added Node Templates does work though. The provided labels are being correctly applied to the created Hetzner Servers.
It's definitely kind of a bummer that the version upgrade effectively corrupted the cluster, but since we're still in trial & setup mode it's not that big of a deal for us.
Hetzner Server Labels seem to work, so I'm happy :)
Thanks for your work in enabling that! @NotANormalNerd

I have the same problem, but in a production setup. Do I have any chance to solve the problem? It appears that the existing cluster is using the old driver and only new clusters use the upgraded driver.

I'm afraid I don't know of any other solution than setting up a fresh cluster. Maybe you could try duplicating your existing cluster and only once everything is up and running switching the DNS records (so you have limited / no downtime)?

NotANormalNerd · 2021-03-01T18:53:49Z

@notitiatech @KochMario A solution for you.

You can clone the repository, checkout the commit before we changed stuff, follow the https://github.com/mxschmitt/ui-driver-hetzner#building and deploy that 'dist' to a webserver of your choice that is reachable by your rancher instance.
Then exchange the "Custom UI URL" with your own and voila, you have another version of the UI, that also does not change as long as you don't update the UI.

On the other hand I don't understand what the problem is with upgrading the driver, also we already fixed that problem as far as I know. So you probably have another problem altogether.

Best regards,
Dennis

mxschmitt · 2021-03-02T09:45:03Z

lets maybe revert that change so the customers can continue to use this ui driver? @NotANormalNerd

NotANormalNerd · 2021-03-02T15:42:20Z

@mxschmitt That probably would break our setup somehow, we could at least not update any node templates in the forseeable future since we rely on the hetzner labels for load balancing.

You would also have to roll forward and empty or the delete the Config.serverLabel setting, since that is saved by rancher and will be still applied. So another code change for that.
I don't support that. Yes that setup got broken, but in my opinion, the solution is to update the hetzner Node driver instead of reverting changes here.

I also haven't seen any "debug" of what actually broke, just This supposedly wasn't compatible... and It appears that....
@KochMario @notitiatech What Version of the Hetzner Node Driver are you running? What Version of Rancher?
Replacing the Download URL with https://github.com/JonasProgrammer/docker-machine-driver-hetzner/releases/download/3.2.0/docker-machine-driver-hetzner_3.2.0_linux_amd64.tar.gz and resaving the node templates should fix the setup and not actually have any other side effects.

Don't get me wrong, I actually believe this could have something to do with the UI driver, but I don't see a reason why the UI driver should roll anywhere.

Also It appears that the existing cluster is using the old driver and only new clusters use the upgraded driver. is not correct in my experience as I upgraded our staging and production cluster that way.

All in all it would be a good idea to version the releases here, so situations like this can be avoided.

It comes down to Linus Torvalds "Don't break userspace" vs Zuckerbergs "Move fast and break things"

ssimeth · 2021-03-02T21:26:56Z

@NotANormalNerd I replaced the URL of the driver with the new one https://github.com/JonasProgrammer/docker-machine-driver-hetzner/releases/download/3.2.0/docker-machine-driver-hetzner_3.2.0_linux_amd64.tar.gz, resaved the node template and ask Rancher to deploy a new worker node from this template. This ended up with the following error Flag provided but not defined: -hetzner-networks; Timeout waiting for ssh key. Deploying a new cluster with the template works fine.

Rancher Version: 2.5.1
Kubernetes Version: 1.18.8-rancher1-1
Docker Version: 19.3.15

NotANormalNerd · 2021-03-03T08:36:46Z

@notitiatech Can you go into your rancher installation and activate debug logging? https://rancher.com/docs/rancher/v2.x/en/troubleshooting/logging/

Then start a new node and see what the debug log is? Specifically it should be something along rancher-machine or deployment.

Also on your text the error is -hetzner-networks in the image it is -hetzner-labels.
I can't really be that it uses the "old driver" for old cluster and the "new driver" for new clusters, since the node drivers are global settings, as well as the node templates. So somewhere in your rancher installation there is a state mismatch of all of that.

ssimeth · 2021-03-03T15:55:49Z

@NotANormalNerd the error message differs from deployment to deployment.

Rancher logs:

2021/03/03 15:50:59 [INFO] Creating jail for c-pm7c9
2021/03/03 15:50:59 [INFO] Provisioning node nt-prod-big3
Incorrect Usage.

Usage: docker-machine create [OPTIONS] [arg...]

Create a machine

Description:
Run 'rancher-machine create --driver name --help' to include the create flags for that driver in the help text.

Options:

--driver, -d "virtualbox" Driver to create machine with. [$MACHINE_DRIVER]
--engine-env [--engine-env option --engine-env option] Specify environment variables to set in the engine
--engine-insecure-registry [--engine-insecure-registry option --engine-insecure-registry option] Specify insecure registries to allow with the created engine
--engine-install-url "https://get.docker.com" Custom URL to use for engine installation [$MACHINE_DOCKER_INSTALL_URL]
--engine-label [--engine-label option --engine-label option] Specify labels for the created engine
--engine-opt [--engine-opt option --engine-opt option] Specify arbitrary flags to include with the created engine in the form flag=value
--engine-registry-mirror [--engine-registry-mirror option --engine-registry-mirror option] Specify registry mirrors to use [$ENGINE_REGISTRY_MIRROR]
--engine-storage-driver Specify a storage driver to use with the engine
--hetzner-api-token Project-specific Hetzner API token [$HETZNER_API_TOKEN]
--hetzner-existing-key-id "0" Existing key ID to use for server; requires --hetzner-existing-key-path [$HETZNER_EXISTING_KEY_ID]
--hetzner-existing-key-path Path to existing key (new public key will be created unless --hetzner-existing-key-id is specified) [$HETZNER_EXISTING_KEY_PATH]
--hetzner-image "ubuntu-16.04" Image to use for server creation [$HETZNER_IMAGE]
--hetzner-image-id "0" Image to use for server creation [$HETZNER_IMAGE_ID]
--hetzner-server-location Location to create machine at [$HETZNER_LOCATION]
--hetzner-server-type "cx11" Server type to create [$HETZNER_TYPE]
--hetzner-user-data Cloud-init based User data [$HETZNER_USER_DATA]
--swarm Configure Machine to join a Swarm cluster
--swarm-addr addr to advertise for Swarm (default: detect and use the machine IP)
--swarm-discovery Discovery service to use with Swarm
--swarm-experimental Enable Swarm experimental features
--swarm-host "tcp://0.0.0.0:3376" ip/socket to listen on for Swarm master
--swarm-image "swarm:latest" Specify Docker image to use for Swarm [$MACHINE_SWARM_IMAGE]
--swarm-join-opt [--swarm-join-opt option --swarm-join-opt option] Define arbitrary flags for Swarm join
--swarm-master Configure Machine to be a Swarm master
--swarm-opt [--swarm-opt option --swarm-opt option] Define arbitrary flags for Swarm master
--swarm-strategy "spread" Define a default scheduling strategy for Swarm
--tls-san [--tls-san option --tls-san option] Support extra SANs for TLS certs
node config nt-prod-big3
is deprecated in v1.19+
handler node-controller: flag provided but not defined: -hetzner-use-private-network, requeuing
Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
is deprecated in v1.19+
is deprecated in v1.19+
is deprecated in v1.19+
is deprecated in v1.19+

NotANormalNerd · 2021-03-03T17:20:19Z

flag provided but not defined: -hetzner-use-private-network, requeuing

It seems you are still running the 2.0.0 release of the Hetzner Driver, since it does not provide the -hetzner-use-private-network: https://github.com/JonasProgrammer/docker-machine-driver-hetzner/releases/tag/2.1.0

When you change the Download URL and save the node driver, does it show in the UI as downloading the driver?

ssimeth · 2021-03-04T11:29:55Z

@NotANormalNerd this is exactly what I thought/said.

When you change the Download URL and save the node driver, does it show in the UI as downloading the driver?

yes

As I said deploying a new Cluster works as expected.

NotANormalNerd · 2021-03-04T15:35:11Z

Okay, there is something I can't wrap my head around and this is clearly off-topic here, as this is not the fault of the UI Component here:

You tell me: You have one rancher instance, which uses different versions of a node driver on different clusters created in the same rancher instance? Even tough you only have one hetzner node driver and this is a global rancher object?

ssimeth · 2021-03-05T08:46:47Z

@NotANormalNerd yes it probably looks like this. I have one Rancher instance with one Hetzner-driver as a global object. Using it in the existing cluster failed with the mentioned errors. Using it with a new cluster works.

NotANormalNerd · 2021-03-05T10:28:14Z

Well I guess you probably should open an issue at https://github.com/rancher/rancher or go ask on their slack. Because this is clearly not a problem with this UI Component.

I also advise you to update your rancher instance to 2.5.5 just to make sure, that's not a bug already fixed.

Also add labels to Hetzner Server from NodeTemplate

6722d6e

mxschmitt reviewed Jan 10, 2021

View reviewed changes

mxschmitt merged commit 988200f into mxschmitt:master Jan 10, 2021

felixpie mentioned this pull request Feb 25, 2021

Bug in new feature "possibility to add additional ssh keys" #112

Open

mxschmitt mentioned this pull request Jul 15, 2021

Question: how are server labels supported? #100

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Also add labels to Hetzner Server from NodeTemplate #109

Also add labels to Hetzner Server from NodeTemplate #109

NotANormalNerd commented Jan 9, 2021

mxschmitt left a comment

NotANormalNerd commented Jan 10, 2021

mxschmitt commented Jan 10, 2021

KochMario commented Jan 12, 2021

NotANormalNerd commented Jan 13, 2021 •

edited

Loading

KochMario commented Jan 14, 2021

mxschmitt commented Jan 14, 2021

NotANormalNerd commented Jan 14, 2021

ssimeth commented Feb 26, 2021 •

edited

Loading

KochMario commented Feb 28, 2021

NotANormalNerd commented Mar 1, 2021 •

edited

Loading

mxschmitt commented Mar 2, 2021

NotANormalNerd commented Mar 2, 2021

ssimeth commented Mar 2, 2021

NotANormalNerd commented Mar 3, 2021

ssimeth commented Mar 3, 2021

NotANormalNerd commented Mar 3, 2021

ssimeth commented Mar 4, 2021 •

edited

Loading

NotANormalNerd commented Mar 4, 2021

ssimeth commented Mar 5, 2021

NotANormalNerd commented Mar 5, 2021

Also add labels to Hetzner Server from NodeTemplate #109

Also add labels to Hetzner Server from NodeTemplate #109

Conversation

NotANormalNerd commented Jan 9, 2021

mxschmitt left a comment

Choose a reason for hiding this comment

NotANormalNerd commented Jan 10, 2021

mxschmitt commented Jan 10, 2021

KochMario commented Jan 12, 2021

NotANormalNerd commented Jan 13, 2021 • edited Loading

KochMario commented Jan 14, 2021

mxschmitt commented Jan 14, 2021

NotANormalNerd commented Jan 14, 2021

ssimeth commented Feb 26, 2021 • edited Loading

KochMario commented Feb 28, 2021

NotANormalNerd commented Mar 1, 2021 • edited Loading

mxschmitt commented Mar 2, 2021

NotANormalNerd commented Mar 2, 2021

ssimeth commented Mar 2, 2021

NotANormalNerd commented Mar 3, 2021

ssimeth commented Mar 3, 2021

NotANormalNerd commented Mar 3, 2021

ssimeth commented Mar 4, 2021 • edited Loading

NotANormalNerd commented Mar 4, 2021

ssimeth commented Mar 5, 2021

NotANormalNerd commented Mar 5, 2021

NotANormalNerd commented Jan 13, 2021 •

edited

Loading

ssimeth commented Feb 26, 2021 •

edited

Loading

NotANormalNerd commented Mar 1, 2021 •

edited

Loading

ssimeth commented Mar 4, 2021 •

edited

Loading