The loss cannot converge when finetuning Llama2-7b-GPTQ on 4090

[cyita]

I finetuned https://huggingface.co/TheBloke/Llama-2-7B-GPTQ on 4090 using the code from this repo and modified the group_size in peft_utils.py, but it seems cannot converge.

Only pass the learning rate = 3e-05 to qalora.py

[StiphyJay]

I finetuned https://huggingface.co/TheBloke/Llama-2-7B-GPTQ on 4090 using the code from this repo and modified the group_size in peft_utils.py, but it seems cannot converge.

Only pass the learning rate = 3e-05 to qalora.py

The same problem. Do you solve this problem?

[duany049]

The same problem. Do you solve this problem?

The same problem. Do you solve this problem?

[duany049]

any update?

[StiphyJay]

any update?

didn't fix.

[duany049]

any update?

didn't fix.

我用W4G32，跟论文一样的配置去训练模型，结果依然不收敛。
你有试过用loss比较大的模型，跑评测么？ 若有，效果如何？

[duany049]

any update?

didn't fix.

老哥，若不好解决，咱可以换一个方案。 你有试到过 2bit量化效果比较好的方案么？

[StiphyJay]

any update?

didn't fix.

老哥，若不好解决，咱可以换一个方案。 你有试到过 2bit量化效果比较好的方案么？

please refer PB-LLM.

[duany049]

any update?

didn't fix.

老哥，若不好解决，咱可以换一个方案。 你有试到过 2bit量化效果比较好的方案么？

please refer PB-LLM.

老哥，我已经解决这个问题了，需要替换auto-gptq中的peft-utils.py文件！

另外，PB-LLM中的QAT方法，你训练时怎么避免内存溢出呢？

[duany049]

I have solved this problem by replacing path/auto_gptq/utils/peft_utils.py by the peft_utils.py file in the project

[duany049]

any update?

didn't fix.

老哥，若不好解决，咱可以换一个方案。 你有试到过 2bit量化效果比较好的方案么？

please refer PB-LLM.

老哥，我已经解决这个问题了，需要替换auto-gptq中的peft-utils.py文件！

另外，PB-LLM中的QAT方法，你训练时怎么避免内存溢出呢？

显存溢出，我80G的显存都不够用！

[wenjingk-xilinx]

Hi @duany049 @cyita @StiphyJay @yuhuixu1993 , could you help to have a look on issue, have you met similar problem? Many thanks!