Skip to content

Integration

Jump to bottom
Ilya Baldin edited this page Jul 29, 2024 · 27 revisions

Integration with EJFAT dataplane and control plane

Overview

E2SAR uses a combination of UDP packets and gRPC+TLS to communicate with loadbalancer control plane and dataplane.

All interactions are captured in the figure below.

The process of using the load balancer begins with (1) reserving a load balancer using an out-of-band information like the admin token and the ejfat URL of the Control Plane agent. This can be done by one of the receivers or by Workflow Management System (WMS) or its proxy.

Once the load balancer is reserved and the details of the reservation communicated out of band (shown with dotted lines) to the worker nodes, they can register themselves with the control plane (2). Alternatively the registration can be peformed by WMS or its proxy assuming the details of the workers are known to it.

At this point (after all workers have been registered and the details of the load balancer communicated to the sender(s) - shown with dotted lines) data can begin traversing the dataplane in segments (3a and 3b).

Periodic sync messages (4) are required to be sent by the sender once a second. Similarly the receiving nodes or WMS on their behalf must periodically use SendState gRPC call (5) to update queue occupancy information in the control plane.

After the workflow completes, the workers can be deregistered (6) - themselves or via WMS. The load balancer can be freed (7).

E2SAR and Control Plane Messaging

Timings and other message rules

EJFAT Control plane has a few timing rules and best practices for implementation:

  • Sync messages should be sent about 1 time per second by the sender (can be more often). The control plane uses a ringbuffer to calculate the rate slope. E2SAR also includes a low-pass filter for this purpose (you can set the sync period and the number of sync periods over which the reported rate is computed).
  • After a worker is registered, sendState must be sent within 10 seconds. In general a period of 100ms is recommended for doing sendState
  • When reserving a load balancer with reserveLB if the passed in time is 0 (i.e. UNIX epoc) the reservation never expires.

Additional Information

Additional details are contained in this document

Clone this wiki locally