Feature Request for MultiProcessExecutor

[Schteve-Earl]

Hey ya'll,
I've recently run into some fundamental issues regarding the use of the MultiThreadedExecutor. In my project, I have multiple services that I need to call concurrently. I've designed my system such that only one service node of each type needs to be launched, and that service uses a multi-threaded executor to handle a bunch of CPU processing tasks in parallel. This paradigm works fine for the nodes that I've written in CPP, but they are being bottle-necked by the nodes that I've written in Python. I understand the rationale in implementing the threading library in the MultiThreadedExecutor, and I think it still has a place for when a node has multiple I/O bound processes without much CPU-based processing. But, in this instance, it would be nice to have an executor that mimics the functionality of the CPP MultiThreadedExecutor, and allows the node to take full advantage of modern processors to parallelize work that is CPU intensive, without having to rewrite the node (and supporting libraries) in CPP. Therefore, I propose the creation of a MultiProcessExecutor. This new type would mimic the functionality of the MultiThreadedExecutor, but would replace the usage of the concurrent.futures ThreadPoolExecutor with the ProcessPoolExecutor. Obviously, care would needed to be taken to ensure that creating new memory spaces doesn't nuke the user's machine. Additionally, I haven't dug too far into the code yet, so I don't know if inter-process communication needs to occur, and how difficult this would be to add to the context. Is this something that the community would like to see?

[clalancette]

I think it would probably be pretty difficult to do, because you would have to share a ton of state between the various processes. That said, if you are interested in giving it a shot, it would be interesting to see what it looks like.

We won't have time to work on it, but I'll leave this open in case you want to leave updates.

[ruthvik92]

I think it would probably be pretty difficult to do, because you would have to share a ton of state between the various processes. That said, if you are interested in giving it a shot, it would be interesting to see what it looks like.

We won't have time to work on it, but I'll leave this open in case you want to leave updates.

I am curious, do you know what are some of the major objects that need to be shared between processes ?