Refactor LocalRuntime, Remove Serialization of Private Attributes, and Collaborator, and Aggregator as a Stateful Actor

[ParthM-GitHub]

Summary of changes:

• participants.py

Changes to support Collaborator, and Aggregator as a stateful actor
Changes to avoid copying of collaborator or aggregator private attributes in every step.
Added GPU / CPU support for collaborator, and aggregator.

• placement.py:

RayExecutor class removed from placement.py as it is not a decorator

• local_runtime.py:

RayExecutor is in local_runtime.py and it's usage is in LocalRuntime
Changes made to avoid recursion
Changes made to avoid serialization of collaborator or aggregator private attributes

Merged remove-torch-dependency with this branch

• Removing torch dependency from get_number_of_gpus function
• Added Keras MNIST Example
• Changed protobuf-3.19.6 to protobuf-3.20.3 in setup.py as per to requirements of TensorFlow
• Added a dummy flush method in stream_redirect.py -> class RedirectStdStream(object)

Verification performed:

• Verified ray memory usage
• Verified checkpoints manually
• Verified with all testcases, and examples available for LocalRuntime

[psfoley]

@ParthM-GitHub I think there's an opportunity here to limit the simultaneous processes and prevent the GPU OOM errors. There's a pattern of using ray.wait(), and we could apply this based on the number of available GPUs in the system - i.e.:

MAX_GPUs = get_number_of_gpus() 
result_refs = []
for _ in range(ALL_COLLABORATORS):
    if len(result_refs) > MAX_GPUs:
        # update result_refs to only
        # track the remaining tasks.
        ready_refs, result_refs = ray.wait(result_refs, num_returns=1)
        ray.get(ready_refs)

    result_refs.append(actor.heavy_compute.remote())

ray.get(result_refs)

This would require us to set the number of gpus per actor on behalf of the user to an appropriate fraction such that each collaborator / aggregator actor can start, and then we would control the scheduling. The downside here is every actor would have access to only one GPU, but that is better than frequent memory errors IMO. And for advanced users who need multiple GPUs per actor, we could still provide an override with a warning that OOM memory errors may be encountered.

[ParthMandaliya]

@psfoley As discussed our understanding is that limiting the number of simultaneous processes, in this case, methods of Ray actor scheduled simultaneously, is unlikely to prevent OOM errors

Our experiments demonstrated that Ray Actors are long-lived components and do not release the memory till the Actor goes out of scope.

As per our understanding, attempting to limit the simultaneous processes by
a) Creating a limited number of Ray Actors and
b) Deleting the Ray actors once their processing is finished
Will be quite inefficient and counterproductive to the reason why Ray actors were implemented in the first place.

[psfoley]

This PR was brought into #910 and merged