Skip to content

Latest commit

 

History

History
71 lines (59 loc) · 3.81 KB

nccl-over-mscclpp.md

File metadata and controls

71 lines (59 loc) · 3.81 KB
Raw

NCCL Over MSCCL++

(limitations)=

Limitations

Current NCCL over MSCCL++ has a few limitations.

  • We do not cover all APIs yet. See the API Support Table for details.
  • Multi-node communication is not supported yet.
  • Currently, collective communication functions may not work correctly if the buffer address is differed from that of previous function calls while sharing the same base address (returned by cuMemGetAddressRange) with the previous address. This is because the current implementation performs zero-copy communication over user buffers, and it is difficult to efficiently inform all ranks if the buffer address dynamically changes.

(api-support-table)=

API Support Table

The table below lists all NCCL APIs (v2.21). We may cover more APIs in the future.

API Name Supported
ncclGetLastError X
ncclGetErrorString O
ncclGetVersion O
ncclGetUniqueId O
ncclCommInitRank O
ncclCommInitAll X
ncclCommInitRankConfig X
ncclCommSplit X
ncclCommFinalize O
ncclCommDestroy O
ncclCommAbort X
ncclCommGetAsyncError O
ncclCommCount O
ncclCommCuDevice O
ncclCommUserRank O
ncclCommRegister X
ncclCommDeregister X
ncclMemAlloc X
ncclMemFree X
ncclAllReduce O
ncclBroadcast X
ncclReduce X
ncclAllGather O
ncclReduceScatter X
ncclGroupStart O
ncclGroupEnd O
ncclSend X
ncclRecv X
ncclRedOpCreatePreMulSum X
ncclRedOpDestroy X

Executor Support

The executor is a versatile tool designed to specify how mscclpp executes algorithms. Currently, only the allReduce operation allows for algorithm customization. The following environment variables can be managed:

  • ALLREDUCEPKT_IP_JSON_FILE: Specifies the path to the JSON file that defines the algorithm for small-sized, in-place operations.
  • ALLREDUCEPKT_OP_JSON_FILE: Specifies the path to the JSON file that defines the algorithm for small-sized, out-of-place operations.
  • ALLREDUCE_IP_JSON_FILE: Specifies the path to the JSON file that defines the algorithm for larger-sized, in-place operations.
  • ALLREDUCE_OP_JSON_FILE: Specifies the path to the JSON file that defines the algorithm for larger-sized, out-of-place operations.
  • ALLREDUCE_SMALL_MSG_BOUNDARY: Defines the size threshold at which the algorithm will switch between fallback code and the customized algorithm for small messages.
  • ALLREDUCE_LARGE_MSG_BOUNDARY: Defines the size threshold at which the algorithm will switch between the customized algorithm for small messages and that for larger messages.
:name: MMSCCL++ Abstractions
:alt: MSCCL++ Abstractions
:align: center

Decision Flowchart for Message Size-Based Algorithm Execution

This is an example of executing the interface with the executor:

mpirun -np 8 -x ALLREDUCEPKT_IP_JSON_FILE=/root/azure-mscclpp/nccl/test/execution-files/allreducepacket.json -x ALLREDUCE_IP_JSON_FILE=/root/azure-mscclpp/nccl/test/execution-files/allreducesm.json -x ALLREDUCE_SMALL_MSG_BOUNDARY=16K -x ALLREDUCE_LARGE_MSG_BOUNDARY=1M ./apps/nccl/test/nccl_api_test