Failed to watch secrets: Stream Error

[shay-ul]

Describe the bug

We have a cluster with more than 10,000 secrets. Kubernetes replicator fails to replicate new secrets, and logs the following:

I0110 13:29:37.790193       1 trace.go:219] Trace[1458323237]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169 (10-Jan-2024 13:28:37.685) (total time: 60104ms):
Trace[1458323237]: ---"Objects listed" error:stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 31; INTERNAL_ERROR; received from peer 60104ms (13:29:37.790)
Trace[1458323237]: [1m0.104611175s] [1m0.104611175s] END
E0110 13:29:37.790213       1 reflector.go:140] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 31; INTERNAL_ERROR; received from peer

Environment:

• Kubernetes version: EKS 1.24
• kubernetes-replicator version: v2.8.0

Additional context
We believe this is a timeout related issue. The API Server takes too long to respond (listing secrets across all namespace). If this is the case, we would like to be able to set a timeout for the secret list commands.
We also set higher CPU and Memory requests and limits. We are still facing this issue even when setting excessive requests and limits.