Implementations of Watch that can be used for most storage layers

[Iceber]

What would you like to be added?

We currently have Watch capability in memory storage, but it requires the binding-apiserver command to run both Clusterpedia APIServer and Clusterpedia Clustersynchro Manager, which prevents us from scaling the service horizontally

We wanted to find a generic Watch mechanism that could be reused on most storage tiers and that would not affect the horizontal scaling of the Clusterpedia APIServer.

For the Watch feature, there are roughly three ways to implement it:

1. Customized to a specific storage layer, using a storage layer with its own event notifications, such as memory, etcd-based Watch, or redis publish-and-subscribe for Watch
   But these specific storage layers may not be good for resource retrieval, and we will lose some performance or functionality for complex resource retrieval

2. A middleware-based subscription publishing for messages, decoupling Query/Watch(APIServer) and collecting resources through a message middleware.
   But the introduction of middleware will increase the complexity of the architecture and may be troublesome for Watch handling when the middleware is down

3. Forwarding Watch requests to the collection of resources -- ClusterSynchroManager
   But the ClusterSynchroManager may need to increase the cache of some events, which inadvertently brings uncontrollable memory pressure
   In addition, the future Agent mode will add a resource collector component that can be scaled horizontally, which will not be able to forward Watch requests correctly

Why is this needed?

#452

[clusterpedia-bot]

Hi @Iceber,
Thanks for opening an issue!
We will look into it as soon as possible.

Details

Instructions for interacting with me using comments are available here.
If you have questions or suggestions related to my behavior, please file an issue against the gh-ci-bot repository.

[xyz2277]

hi, iceber, this is my solution for the new Watch feature.

[xyz2277]

1. A message queue is used for transporting events from ClusterSynchroManager to ApiServer, before transporting, events must be persisted to the storage layer, and rv of events must be converted to multi-clusters-rv.
2. EventCache is a buffer for events, when watch request is arrived, initialization events will be found in it, and the following events will be coming from the Message queue.
3. Multi-clusters-rv will be returned in a list request.

[xyz2277]

What do you think of it. @Iceber

[Iceber]

With the addition of the new component, I think one issue we can't ignore is the behavior of the Synchor and Apiserver after the message queue hangs