Is it possible to use an open source model that Huggingface has quantified？

[wangxiaochun520]

Personally, I would like to fine-tune with Huggingface's models that have been quantized by int4, but currently your code needs GPTQ local group-wise quantization before you can use QA-lora when fine-tuning, which feels that this limits some of the capabilities of your project.

[yuhuixu1993]

Personally, I would like to fine-tune with Huggingface's models that have been quantized by int4, but currently your code needs GPTQ local group-wise quantization before you can use QA-lora when fine-tuning, which feels that this limits some of the capabilities of your project.

Thanks for your suggestion. I would consider in the future.