KokkosComm backends

[dutkalex]

Hi folks!
KokkosComm is still a very young project, and the main target for the immediate future is to get down a first meaningful implementation based on MPI to test, refine and validate the core ideas. However, as already hinted in other threads, MPI will probably not be the one and only communication model that we support in the long run, similarly to what KokkosCore does.
IMO the first alternative that we should consider implementing, is the dummy serial mode, which just does... well, nothing. It may seem useless at first sight, but being able to gracefully disable communications is very useful in practice, because the client code can run on a small laptop as well as on a massive supercomputer out of the box, without changing a single line of code. This would be a good first step towards a multi-backend architecture because implementing this is trivial, and it would let us focus on the architectural aspects of the question. I opened this thread to initiate a discussion on this topic of supporting different backends, gather ideas on how to approach the problem, what other backends would we like to support, the semantic implications etc.
Please feel free to suggest any idea that could help achieve this!
Best regards,
Alex

[dutkalex]

The first milestone is be to come up with a working API which can:

• Delegate communications to MPI
• Work in serial mode
• Delegate to NCCL

Some questions in bulk regarding the communication semantics and what we want to support:

• For sure: usual point to point send/recv and collective communications (blocking and async modes)
• What about one-sided/RMA communications?
• What about device-initiated communications?
• What does NCCL not support compared to MPI? What about other long-term backend targets? Can we use MPI as a fallback for what is not supported?
• Should we restrict the API to what is supported by all backends? Are there cases where we should emulate what is not supported by all backends instead?

[cedricchevalier19]

Thanks for all these ideas @dutkalex.

For sure: usual point to point send/recv and collective communications (blocking and async modes)

I am not even sure that we really need the blocking ones.

What about one-sided/RMA communications?

For me, we are about explicit communications. They can be one-sided but not "transparent". For now, Kokkos Remote Spaces is a better solution for RMA.

What about device-initiated communications?

We want to support that. It should be possible to initiate communications from a parallel region.

What does NCCL not support compared to MPI? What about other long-term backend targets? Can we use MPI as a fallback for what is not supported?

NCCL has a simpler semantic than MPI. It would be better to think the other way around: try to keep the interface as small as possible so that it can map directly to the backend.

However, for now, the only priority is MPI. We want to provide a robust wrapper on a few MPI functions with no overhead compared to an ad-hoc code.

[dutkalex]

For me, we are about explicit communications. They can be one-sided but not "transparent". For now, Kokkos Remote Spaces is a better solution for RMA.

I don't get why you say that this flavor of MPI calls are implicit communications. I think there is a nuance with what RemoteSpaces does. Some problems are truly simpler to express correctly and require much less boilerplate code with one-sided calls. Once again, I am not talking about using global distributed arrays and having the communications be implicit...

[dutkalex]

I am not even sure that we really need the blocking ones.

Can you elaborate?
Do you say that because they can easily be emulated with async calls + directly waiting on the request?
Or because blocking communications can lead to deadlocks if issues inside of parallel kernels?

[devreal]

Work in serial/no-op mode (should be easy)

A word of caution here. You need to implement tag matching, ordering, and type conversion for send/recv on the same process (on potentially different streams / threads) and reductions on all supported operators/types. It sounds like a low hanging fruit to provide empty operations and I understand the appeal but that doesn't meet the semantics we already have.

IMO, having an MPI and NCCL backend will be sufficient. Either one of them will be available or can be made available easily.

[cedricchevalier19]

I agree with @devreal, I do not think a serial backend is very useful. It is hard to implement correctly, and MPI implementations are better than ours.
BTW, some MPI-only programs use serial MPI implementation such as siMPI to provide a sequential version that does not require MPI (for example, https://github.com/trilinos/Trilinos/tree/master/packages/zoltan/siMPI).

[dutkalex]

It sounds like a low hanging fruit to provide empty operations and I understand the appeal but that doesn't meet the semantics we already have.

I don't get where's the harm as long as the behavior is well documented, as it seems quite natural that asking for communications between processes in a single-process context should be a no-op.

[devreal]

There are cases where it is natural to express communication regardless of whether that happens on the same process or not:

int left_neighbor = (comm_rank + comm_size - 1) % comm_size;
int right_neighbor = (comm_rank + 1) % comm_size;
// left_neighbor == right_neighbor == comm_rank for single-process
KokkosComm::isend(..., left_neighbor);
KokkosComm::irecv(..., right_neighbor);

If we disallow that for some backends then a) KokkosComm is not portable, and b) developers must code separate code paths to catch the single-process / funny backend case.

[devreal]

Maybe KokkosComm can ship siMPI as a fallback if no other multi-process backend is functional. KokkosComm doesn't necessarily have to provide its own implementation.

[dutkalex]

There are cases where it is natural to express communication regardless of whether that happens on the same process or not

The argument makes sense

Maybe KokkosComm can ship siMPI as a fallback if no other multi-process backend is functional. KokkosComm doesn't necessarily have to provide its own implementation

Indeed, if good single-process implementations already exist this seems like a good solution

[janciesko]

@nicoleavans