RuntimeError: value cannot be converted to type at::Half without overflow

[jeffxtang]

I followed the fine-tune-code-llama.ipynb and got an error when running trainer.train():

File ~/anaconda3/envs/codallama/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:50, in make_causal_mask(input_ids_shape, dtype, device, past_key_values_length)
46 """
47 Make causal mask used for bi-directional self-attention.
48 """
49 bsz, tgt_len = input_ids_shape
---> 50 mask = torch.full((tgt_len, tgt_len), torch.finfo(dtype).min, device=device)
51 mask_cond = torch.arange(mask.size(-1), device=device)
52 mask.masked_fill(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)

RuntimeError: value cannot be converted to type at::Half without overflow

My transformers version is 4.34.0.dev0 and accelerate is >=0.23.0. Any ideas? Thanks!