TL/CUDA: Linear Broadcast for GPU

[ikryukov]

What

Linear CUDA Broadcast implementation with Active set feature support.

Why ?

• Functional improvement, parity with others communication libraries.
• Ability to place many ranks on single GPU
• No GPU blocking, communication initiated from host
• Active set might be used to emulate P2P send/receive on top of broadcast collective

How ?

Naive approach where root rank writes data to own shared buffer and others ranks read from it through NVLink.
The algorithm enables the initiation of multiple simultaneous peer-to-peer (P2P) communications, constrained only by user-defined tags. During its execution, the algorithm identifies the first available free barrier and acquires it. This barrier, along with associated resources, is then utilized as a scratch buffer to facilitate efficient operations.

[ikryukov]

Configuration string:
./configure --with-ucx=$HPCX_UCX_DIR --with-cuda=/usr/local/cuda --with-mpi=$HPCX_MPI_DIR --enable-gtest --prefix=$PWD/install --with-nvcc-gencode="-gencode=arch=compute_80,code=sm_80" --enable-debug
Run string:
mpirun --mca coll ^hcoll --mca coll_ucc_enable 0 -x LD_LIBRARY_PATH=/home/ikryukov/work/ucc/install/lib:$LD_LIBRARY_PATH -x UCC_CLS=basic -x UCC_TLS=ucp,cuda -x xUCC_LOG_LEVEL=info -x UCC_TL_CUDA_LOG_LEVEL=debug -x UCC_LOG_LEVEL=info -x UCC_CONFIG_FILE= -np 2 ./install/bin/ucc_test_mpi -c bcast --teams world -M cuda -O 0 -S 2

[swx-jenkins3]

Can one of the admins verify this patch?

[samnordmann]

Looks good to me! Thanks!
I only left some minor remarks. Can you, in addition, add this algo to the tests?

[ikryukov]

Looks good to me! Thanks! I only left some minor remarks. Can you, in addition, add this algo to the tests?

Thanks for review, addressed comments and added test to validate bcast for cuda too.

[janjust]

looks good

[manjugv]

ping @Sergei-Lebedev

[samnordmann]

ok to test