Expert deepcopy raises PickleError

[sxontheway]

  File "Megatron-DeepSpeed-main/megatron/core/transformer/deepspeed_moe/experts.py", line 15, in <listcomp>
    self.deepspeed_experts = torch.nn.ModuleList([copy.deepcopy(expert) for i in range(num_local_experts)])
  File "/opt/conda/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/conda/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/opt/conda/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.8/copy.py", line 161, in deepcopy
    rv = reductor(4)
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/_script.py", line 62, in _reduce
    raise pickle.PickleError("ScriptFunction cannot be pickled")

The deepcopy in this line of code self.deepspeed_experts = torch.nn.ModuleList([copy.deepcopy(expert) for i in range(num_local_experts)]) in deepspeed/moe/layer.py raises PickleError. However, removing deepcopy this problems solved.
Is this deepcopy neceaary?

Appreciate for your response!

@arashb @selfReference @AlphardWang @duli2012 @conglongli