Network-transparent channels?

[rcarmo]

I've been mulling the need to coordinate a few processes on different machines.

Is something like this on the roadmap?

Although I use MQTT for cross-language message passing (and am happy with it), it would be interesting if one could set up channels across machines using something that is more popular (say Redis, which is popping up all over the place these days, or 0MQ, which is also fairly agnostic).

[MichaelAz]

Golang has this with dockers libchan (https://github.com/docker/libchan), goless support for this would be really cool. Perhaps before considering network transparent channels, we should think of process transparent ones?

[rgalanakis]

I've considered it but it seems like using multiprocessing and its Queue/Pipes/etc are basically the same thing for process communication. It'd be a quick wrapper and not that useful IMO since I hate dealing with multiprocessing. Its not 'transparent'. Since things will need to be serialized, we'll be limited in what we can send, which is especially important to Go-style stuff because we're often having an anonymous function executing in a goroutine.

That said, I still like the idea if it can be done well. I just don't see how to do it well yet! :) If we could not bother with goless having to manage processes (the caller is responsible for managing them), that'd be fine. In the libchan example, I don't see how to pass data between processes- what are the endpoints, how do you tell data where to go?

[MichaelAz]

The docs aren't great (practically non existant) but from looking at the godoc, they do local rpc over unix-domain sockets by wrapping golangs net.UnixConn and remote rpc over http2 with net.Conn. Anyway, I meant this mainly as an inspiration.

Back to the point, when you put it that way, there's a stable RPC module in the stdlib (multiprocessing) and from some quick googling there's also a mature remote RPC implementation (pyro). So, on second thought, perhaps this is not something we need after all (or we can just wrap those and call it a day, but I don't think it will bring much value)

[rcarmo]

I'm fine with just using ZeroMQ when I want to reach out to another machine (or even another process on the same machine, since at least then I have the option of moving stuff to other boxes without changing my code), and marshaling code and data to send off to an executor is a (somewhat) solved problem (Celery, mincemeat -- it marshals functions and tosses them over the link to an executor), so my question was more along the lines of project scoping.

(Edit: You might want to consider looking at ZeroMQ/zmq anyway, since it also supports in-process communication and might be a suitable alternative to the current implementation of channels)

[MichaelAz]

So, I've been thinking about this and I've got a proposal:

1. Backends that allow for network-transparency will provide a serialization_policy function that, given a channel, returns a serializer for sending data over the channel (this, of course, works for de-serialization as well). So, for example, a ZeroMQ backend could use pickle for inproc sockets and json for tcp sockets. Users will be able to override this with custom serialization logic.
2. Channels must use these serializers if the backend supports them.
3. Network transparent channels will be created with netchan instead of chan and will accept some indication as to where to connect to (a uri for ZeroMQ of nanomsg, for example)

What do you guys think? I'll probably hack on this over the coming week.

[ctismer]

This is very interesting!
It would be perfect to be able to adopt things incrementally.
First, people craft a simple in-process model.
When this works, they add network transparent channels and use ZMQ
for instance.
It was a long missing feature to have trans-machine channels, and maybe goless
fills exactly this gap.

[rgalanakis]

Hmmm, as I'm not experienced enough in this domain (haven't looked at other libraries in detail), I can't offer much. I would love to see some pseudocode even before you go down the prototype route. I have a lot of questions but feel they'd probably just be better answered with some code.

Also of interest for you would be another project of mine: https://github.com/rgalanakis/dccautomation
and particularly
https://github.com/rgalanakis/dccautomation/blob/master/dccautomation/compat.py
Basically that file contains code that abstracts away zmq/nanomsg/fifo into a common interface (that closely mimics zmq).

Ultimately we'd either have to provide 'backends' for things like zmq/nanomsg/snakemq/http/rabbitmq/amqp, or require callers to provide their own (and then create separate packages for each implementation, so people weren't reinventing the wheel). Either way, creating a homogeneous interface across these systems can be extremely tricky, especially if you want "synchronous" channel behavior. You'd probably want to limit the netchan channel behavior to be totally "unbufferred", as bounded queues aren't a universal concept.

Looking forward to seeing some code, or at least an API skeleton!

[rgalanakis]

I just saw that Go had an old netchan package. Worth taking a look (and googling for more info), though it was discontinued.
This is pretty useful too: https://groups.google.com/forum/#!topic/golang-nuts/Er3TetntSmg

[rgalanakis]

So @MichaelAz and @rcarmo you gave me the bug and I wrote something up very quickly while my son is napping: 6c3da3a
Perhaps I'm spoiled by higher-level libraries, but it seems these 30 lines of code do basically what we'd need? Or am I way off (like I said, I don't have much production experience with a use-case for this, so I'm just riffing).