hello, i found a problem about the execution time

[Suetsumu]

I used the single-layer algorithm(By use __merge function to merge every layer) to run this program,then I found that the non-overlapped time is bigger than MG-WFBP algorithm.This is very right. But the time used in every iteration and epoch is less than MG-WFBP.

[shyhuai]

Hi, May I know if you have reset the alpha and beta values (https://github.com/HKBU-HPML/MG-WFBP/blob/master/distributed_optimizer.py#L166) that should be evaluated from your own cluster? MG-WFBP requires this information (can use nccl-tests to estimate) to generate a merging solution that fits the cluster.

[Suetsumu]

thanks for your reply. I will try your suggestion

[Suetsumu]

I use the benchmark function (

MG-WFBP/distributed_optimizer.py

Line 105 in 5b8ad54

def _benchmark_communication(self):

) to get the alpha and beta value and I found that the alpha value is almost 0.002s. The value is so big that the all layer have merge to one signer layer. I don't know why the startup speed is so slow. And I found that NCCL info show that my lab's GPUs don't support P2P.
Is it for this reason？

[shyhuai]

Hi, the startup time (i.e., alpha) is quite large, so MG-WFBP tends to merge all layers into a single one. You may need to check your hardware configuration. There may be different reasons (e.g., p2p support or not, # of CPU PCIe lanes, PCIe version, etc.) causing a large latency.

[Suetsumu]

this is the nccl_tests result that I used 5 NVIDIA GeForce RTX 3090.

[Suetsumu]

OK,I will check my hardware configuration. Thanks for your suggestion