Relation data is not updated when loadbalancer ip address changes

[gboutry]

Bug Description

A user encountered this on a ~40 days old 4 nodes sunbeam installation. The MetalLB assigned address changed, but was not reflected in the deployment.

Status output:

traefik-public/0*            maintenance  idle       10.1.198.155         updating ingress configuration for 'ingress:95'
traefik-public/1             active       idle       10.1.123.97          Serving at 10.206.54.133
traefik-public/2             active       idle       10.1.14.108          Serving at 10.206.54.133
traefik-rgw/0*               active       idle       10.1.198.156         Serving at 10.206.54.122
traefik-rgw/1                active       idle       10.1.181.125         Serving at 10.206.54.122
traefik-rgw/2                active       idle       10.1.14.104          Serving at 10.206.54.122
traefik/0*                   active       idle       10.1.198.152         Serving at 10.206.54.144
traefik/1                    active       idle       10.1.14.113          Serving at 10.206.54.144
traefik/2                    active       idle       10.1.123.96          Serving at 10.206.54.144

OpenStack Endpoint output

node1@node1:~$ openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                                         |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------------------------+
| 0d931254080b436bafbd7af755394c95 | RegionOne | nova         | compute      | True    | admin     | http://10.152.183.109:8774/v2.1                             |
| 1274f2566a4849d49ea78610b4656c97 | RegionOne | glance       | image        | True    | admin     | http://10.152.183.144:9292                                  |
| 23f3f489ba0d4c2584d9abbfc723f6a9 | RegionOne | cinderv2     | volumev2     | True    | admin     | http://10.152.183.115:8776/v2/$(tenant\_id)s                 |
| 24d48cb517204398a34fa8945a622782 | RegionOne | cinderv2     | volumev2     | True    | internal  | http://10.206.54.144:80/openstack-cinder/v2/$(tenant\_id)s |
| 366bdb9559584494ba59008b95afaced | RegionOne | cinderv3     | volumev3     | True    | internal  | http://10.206.54.144:80/openstack-cinder/v3/$(tenant\_id)s |
| 4ef30ffe0a5749e58fe8133abd4950a6 | RegionOne | neutron      | network      | True    | public    | http://10.206.54.122:80/openstack-neutron                 |
| 56c10509bd314a56999353112b2891d4 | RegionOne | s3           | s3           | True    | admin     | http://10.206.54.133                                      |
| 635ca7ad15e34ffb9ba9b67611d15096 | RegionOne | glance       | image        | True    | public    | http://10.206.54.122:80/openstack-glance                  |
| 6cff6cf96b3640aea86aa6a80040f112 | RegionOne | nova         | compute      | True    | internal  | http://10.206.54.144:80/openstack-nova/v2.1               |
| 7a64f3ce29bb40409fed4a181965a156 | RegionOne | swift        | object-store | True    | internal  | http://10.206.54.133/swift/v1/AUTH\_$(project\_id)s         |
| 84d47eb922434f9c88a1e844a84202a4 | RegionOne | keystone     | identity     | True    | public    | http://10.206.54.133/openstack-keystone/v3                |
| 858ceba107b447c6a6ae7413efdff3db | RegionOne | cinderv2     | volumev2     | True    | public    | http://10.206.54.122:80/openstack-cinder/v2/$(tenant\_id)s |
| 8ebdf9847f16412a8aa5d340a7bc823f | RegionOne | placement    | placement    | True    | admin     | http://10.152.183.114:8778                                  |
| 8f8fdf45f28f42c4b591f03f84a31494 | RegionOne | swift        | object-store | True    | admin     | http://10.206.54.133/swift                                |
| 96b2dc3883ba4996b378c0abff1eddf9 | RegionOne | placement    | placement    | True    | internal  | http://10.206.54.144:80/openstack-placement               |
| a0cea06c931e4bc4b50b0a9272849e96 | RegionOne | cinderv3     | volumev3     | True    | admin     | http://10.152.183.115:8776/v3/$(tenant\_id)s                 |
| a1b03ea2dcce46e6879125a36896be44 | RegionOne | glance       | image        | True    | internal  | http://10.206.54.144:80/openstack-glance                  |
| a2a5fadce49149969e33bc28f6dd276f | RegionOne | swift        | object-store | True    | public    | http://10.206.54.133/swift/v1/AUTH\_$(project\_id)s         |
| a743d6f3d5e4442180733ad76eab03e2 | RegionOne | neutron      | network      | True    | internal  | http://10.206.54.144:80/openstack-neutron                 |
| b1e82c0337ea4678a581d5fb149400b3 | RegionOne | keystone     | identity     | True    | internal  | http://10.206.54.144/openstack-keystone/v3                |
| bf4e05368845477b8e79065a690752d8 | RegionOne | s3           | s3           | True    | internal  | http://10.206.54.133                                      |
| c90f23367b3c4d3d9fd15b515d0a9204 | RegionOne | neutron      | network      | True    | admin     | http://10.152.183.186:9696                                  |
| ddf93583646b4eca9b0b1eb63e38492e | RegionOne | placement    | placement    | True    | public    | http://10.206.54.122:80/openstack-placement               |
| e6160ea3f1604eb1b7a234b8844ee021 | RegionOne | cinderv3     | volumev3     | True    | public    | http://10.206.54.122:80/openstack-cinder/v3/$(tenant\_id)s |
| eb1ae57971c74ef89d0e71f38fcadcca | RegionOne | nova         | compute      | True    | public    | http://10.206.54.122:80/openstack-nova/v2.1               |
| f39bf16c4ed94ff19aeb00882a879db5 | RegionOne | keystone     | identity     | True    | admin     | http://10.152.183.139:5000/v3                               |
| fb6e6339536b4b65add958b62e42633b | RegionOne | s3           | s3           | True    | public    | http://10.206.54.133                                      |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------------------------+
node1@node1:~$

To workaround this, I had the user perform the following actions:

# prevent the ip to change
kubectl annotate svc -n openstack traefik-lb metallb.universe.tf/loadBalancerIPs=10.206.54.144
kubectl annotate svc -n openstack traefik-public-lb metallb.universe.tf/loadBalancerIPs=10.206.54.133
kubectl annotate svc -n openstack traefik-rgw-lb metallb.universe.tf/loadBalancerIPs=10.206.54.122
# kill the leaders to force traefik to re-assert itself
kubectl delete -n openstack pod/traefik-0
kubectl delete -n openstack pod/traefik-public-0
kubectl delete -n openstack pod/traefik-rgw-0

To Reproduce

1. juju deploy traefik-k8s traefik --channel latest/stable
3. kubectl annotate svc -n traefik-lb metallb.universe.tf/loadBalancerIPs= (this will force the SVC to change ip address)

Environment

microk8s 1.28-strict
juju 3.5.3
traefik latest/stable 194

Relevant log output

As soon as the leader unit was restarted, it updated all the relation data, and fixed the issue.

Additional context

No response

[dstathis]

This appears to be a dup of #377. Reopen if you disagree.