Skip to content

Commit

Permalink
Update S0-L25.md
Browse files Browse the repository at this point in the history
  • Loading branch information
qiyanjun authored Apr 13, 2024
1 parent 4aff4cc commit c11a2f5
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion _contents/S0-L25.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,11 @@ In this session, our readings cover:

## More readings


### Must know tools for training/finetuning LLM's -
1. Torchtune - Build on top of Pytorch, for training and finetuning LLM's. Uses yaml based configs for easily running experiments. Github - https://lnkd.in/ghu6wx9r
2. axolotl - Built on top on Huggigface peft and transformer library, supports fine-tuning a large number for models like Mistral, LLama etc. Provides support for techniques like RLHF, DPO, LORA, qLORA etc. Github - https://lnkd.in/gYpisva9
3. LitGPT - Build on nanoGPT and Megatron, support pre-training and fine-tuning, has examples like Starcoder, TinyLlama etc. Github - https://lnkd.in/gKisgXms
4. Maxtext - Jax based library for training LLM's on Google TPU's with configs for models like Gemma, Mistral and LLama2 etc. Github - https://lnkd.in/gjeHvZF4

### Instruction Tuning for Large Language Models: A Survey
+ https://arxiv.org/abs/2308.10792
Expand Down

0 comments on commit c11a2f5

Please sign in to comment.