Legion hang on rank 16 on DGX100

[suranap]

Running FlexFlow on Stanford's Marlowe, which is a DGX H100 machine. It completes the entire training run and hangs on Legion shutdown for rank 16.

• 1 node * 4 procs/node = WORKS
• 1 node * 8 procs/node = WORKS
• 2 node * 4 procs/node = WORKS
• 2 node * 8 procs/node = FREEZE
• 4 node * 4 procs/node = FREEZE

Here's my srun:
srun --output ${LOGS}/run_%t.log --ntasks-per-node=8 select_gpu_device.sh ./build-default/flexflow_python examples/python/native/mnist_mlp.py -ll:py 1 -ll:gpu 1 -ll:fsize 70000 -ll:zsize 10000 -lg:inorder -ll:force_kthreads

Attached backtrace and logs for 2 node * 8 procs/node:
N2n16_flexflow.tgz

Let me know what else you need to debug this.

[lightsighter]

Report the output of -level shutdown=2.