Multi Rank Forward Sampling is not Diverse across Ranks

[josephdviviano]

When doing multi-rank training, it appears that the diversity across ranks is lower than it should be when compared with a single rank training with the same global batch size. We would expect performance to be the same but multi rank takes much more time -- it must mean that there are repeats across the ranks.