test_transformer_engine measures SP with overlapping.

[wujingyue]

It doesn't show a clear win for overlapping, but it's worth checking in the benchmark so we have a common ground to start with.

$ mpirun -np 8 --output-filename /tmp/test_transformer_engine pytest tests/python/test_transformer_engine.py --only-mpi

$ cat /tmp/test_transformer_engine/1/rank.0/stdout

------------------------------------------------------------------------------------------------ benchmark: 6 tests ------------------------------------------------------------------------------------------------
Name (time in ms)                                     Min                 Max               Mean             StdDev            Median                IQR            Outliers       OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_transformer_layer[nonoverlap-tp-forward]      2.3807 (1.0)       23.1250 (5.49)      7.6998 (2.24)      8.9569 (68.21)    2.4966 (1.0)       9.3730 (99.94)         1;0  129.8728 (0.45)          5           1
test_transformer_layer[nonoverlap-sp-forward]      2.3829 (1.00)     101.7016 (24.15)    22.7121 (6.60)     44.1618 (336.32)   2.9859 (1.20)     26.0737 (278.02)        1;1   44.0294 (0.15)          5           1
test_transformer_layer[overlap-sp-forward]         3.1758 (1.33)       4.2117 (1.0)       3.4395 (1.0)       0.4338 (3.30)     3.2658 (1.31)      0.2859 (3.05)          1;1  290.7373 (1.0)           5           1
test_transformer_layer[nonoverlap-tp-backward]     4.2101 (1.77)       7.6505 (1.82)      5.0007 (1.45)      1.4866 (11.32)    4.3323 (1.74)      1.0682 (11.39)         1;1  199.9716 (0.69)          5           1
test_transformer_layer[overlap-sp-backward]        4.3271 (1.82)       4.6460 (1.10)      4.4132 (1.28)      0.1313 (1.0)      4.3606 (1.75)      0.0938 (1.0)           1;1  226.5940 (0.78)          5           1
test_transformer_layer[nonoverlap-sp-backward]     4.3753 (1.84)       5.1469 (1.22)      4.5443 (1.32)      0.3372 (2.57)     4.3949 (1.76)      0.2163 (2.31)          1;1  220.0540 (0.76)          5           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[wujingyue]

!test