🌱 Reduce reconcilements

[janiskemper]

What this PR does / why we need it:

• Filter HCloudMachine events so that status updates of the resource don't trigger another reconcilement of HCloudMachine
• Filter HetznerCluster events so that status updates of the resource don't trigger another reconcilement of HetznerCluster
• Filter HetznerCluster events so that some status updates that are not important don't trigger reconcilement of HCloudMachine
• Filter Machine events so that status updates of the resource don't trigger another reconcilement of HCloudMachine a
• Filter Cluster events so that status updates of the resource don't trigger another reconcilement of HetznerCluster
• Increase the time period to requeue if a server is starting
• Increase the time period to requeue if a server has not status yet
• Increase the time period to requeue if the kubeapi server is not yet reachable before adding it as target to the load balancer
• Don't trigger multiple shutdown calls for one server and increase the timeout to requeue after a shutdown is triggered

TODOs:

• squash commits
• include documentation
• add unit tests

Side note:
I will add an issue for unit testing the functions I have added after this is getting merged.

[apricote]

Two notes about the predicates, otherwise this looks good and a step in the right direction. Would love to have some better options for finding/debugging this from controller-runtime.

[guettli]

Two notes about the predicates, otherwise this looks good and a step in the right direction. Would love to have some better options for finding/debugging this from controller-runtime.

@apricote what kind of debugging do you mean here? BTW, you can get the total reconcile counts per CRD:

k port-forward caph-controller-manager-XXX 8080:8080

curl localhost:8080/metrics | grep reconcile_total

controller_runtime_reconcile_total{controller="hcloudmachine",result="error"} 0
controller_runtime_reconcile_total{controller="hcloudmachine",result="requeue"} 0
controller_runtime_reconcile_total{controller="hcloudmachine",result="requeue_after"} 162
controller_runtime_reconcile_total{controller="hcloudmachine",result="success"} 59
...

[apricote]

I would love to have a flag in controller-runtime --debug-controller-reconciles that show why a reconcile was triggered:

• Was it from an event? Was it because of the reconcile interval?
• If event, which event (type, kind, namespacedname)?
• Which predicates allowed this event? Which denied?

Some transparent insights into handler.EnqueueRequestsFromMapFunc(), builder.WithPredicates(), WithEventFilter() and friends.

Not something that you want running in our prod clusters, but something that can be enabled locally (or through some diagnostics API) to figure out why it reconciles more often then expected.

[apricote]

One alternative would be distributed tracing for the Kubernetes API that includes any actions taken by controllers because of watches. But I guess that would be way harder and spans many more components.

[guettli]

@apricote good questions. I have no answer to them, so I asked here to get more insights: https://kubernetes.slack.com/archives/C02MRBMN00Z/p1720524329698439