Secrets not being updated after patch

[ErikTMA]

Describe the bug
We use ArgoCD to deploy applications which have the destination secrets. If a secret is patched through Argo, it is not being updated by the replicator.
ie:
We install a new application - secret is correctly updated by the replicator.
We make an update to the app causing ArgoCD to reconcile, which patches the secret. The secret is not being updated by the replicator.

To Reproduce
Update any destination secret in a pull configuration. ie kubectl apply -f secret.yaml

Expected behavior
I expect the secret to be updated whenever it is patched.

Environment:

• Kubernetes version: 1.22.15-gke.100
• kubernetes-replicator version: 2.7.3

[eugenberend]

@ErikTMA hi, what is your secret type?
I can confirm this issue with kubernetes.io/tls secret type.

[ErikTMA]

@ErikTMA hi, what is your secret type? I can confirm this issue with kubernetes.io/tls secret type.

It's a tls secret as well.

[yevgeniyo-ps]

Seeing the same problem. We are using external secrets for accessing ECR (ECR token TTL is about 12 hours), it means that we have to regenerate token every few hours. We wanted to copy secret with kubernetes-replicator, but we are seeing now that if original secret was updated, cloned ones are not.

[strowi]

Same here, deploying a secret:

apiVersion: v1
data:
  .dockerconfigjson:  e30K
kind: Secret
metadata:
  annotations:
    replicator.v1.mittwald.de/replicate-from: app/registry.gitlab.com
  name: registry.gitlab.com
  namespace: frontend-develop
type: kubernetes.io/dockerconfigjson

The first time works, and gets updated to:

apiVersion: v1
data:
  .dockerconfigjson:  ......
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{".dockerconfigjson":"e30K"},"kind":"Secret","metadata":{"annotations":{"replicator.v1.mittwald.de/replicate-from":"app/registry.gitlab.com"},"name":"registry.gitlab.com","namespace":"frontend-develop"},"typ
e":"kubernetes.io/dockerconfigjson"}
    replicator.v1.mittwald.de/replicate-from: app/registry.gitlab.com
    replicator.v1.mittwald.de/replicated-at: "2024-07-19T14:34:27Z"
    replicator.v1.mittwald.de/replicated-from-version: "3717"
    replicator.v1.mittwald.de/replicated-keys: .dockerconfigjson
  creationTimestamp: "2024-07-18T09:54:35Z"
  name: registry.gitlab.com
  namespace: frontend-develop
  resourceVersion: "1108217"
  uid: 841a62f5-ad03-4472-8dae-63b34cb630d1
type: kubernetes.io/dockerconfigjson

The re-applying it show the first one with some annotations:

apiVersion: v1
data:
  .dockerconfigjson: e30K
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{".dockerconfigjson":"e30K"},"kind":"Secret","metadata":{"annotations":{"replicator.v1.mittwald.de/replicate-from":"app/registry.gitlab.com"},"name":"registry.gitlab.com","namespace":"frontend-develop"},"type":"kubernetes.io/dockerconfigjson"}
    replicator.v1.mittwald.de/replicate-from: app/registry.gitlab.com
    replicator.v1.mittwald.de/replicated-at: "2024-07-19T14:34:27Z"
    replicator.v1.mittwald.de/replicated-from-version: "3717"
    replicator.v1.mittwald.de/replicated-keys: .dockerconfigjson
  creationTimestamp: "2024-07-18T09:54:35Z"
  name: registry.gitlab.com
  namespace: frontend-develop
  resourceVersion: "1108756"
  uid: 841a62f5-ad03-4472-8dae-63b34cb630d1
type: kubernetes.io/dockerconfigjson

Removing the version-annotation, it gets updated correctly again.

Not sure how kubernetes-replicator exactly works, but this might be related to a kubectl's that won't update fields missing the patch..

PS: I might have found a workaround - adding an annotation replicator.v1.mittwald.de/replicated-from-version: '' to the secret on the pull-end will update the secret each time.

[stippi2]

This issue is fixed (for secrets and config maps at least) when you use the new --sync-by-content toggle. More details about the root cause in this issue.

[bygui86]

@stippi2 so enabling --sync-by-content this issue should be solved right?
thanks a lot for support!

[stippi]

It should be, but there is no new release, yet.

[bygui86]

@stippi2 / @stippi ah ok. In fact I upgraded to v2.10.2 and the flag is not recognised :( when do you plan to release it?

[stippi]

I am not a maintainer here, sorry. I was hoping @martin-helmich would trigger a new release. :-)

[martin-helmich]

Yep, hang on... New release is building now. 🙂 ⏳

EDIT:
Here you go: https://github.com/mittwald/kubernetes-replicator/releases/tag/v2.11.0

[stippi]

@bygui86 Once you had a chance to test the new release, please let us know if the issue is fixed for you! :-D

[bygui86]

@stippi thanks a lot!

[bygui86]

@stippi startup is fine, --sync-by-content flag well accepted and a Secret got synced, so I think it works for now :) thanks!