Paper: https://www.zichenz.me/project/slm_tot/slm_tot.pdf
This repository contains the code and datasets used for the paper "Babysitting a Small Language Model through One-Step Tree-of-Thoughts Knowledge Distillation". The project explores a novel approach to enhance the reasoning capabilities of Small Language Models (SLMs) using a simplified prompting method called One-Step Tree-of-Thoughts (ToT) and knowledge distillation from Large Language Models (LLMs).
The project addresses the limitations of SLMs in handling complex reasoning tasks by:
- Introducing the One-Step Tree-of-Thoughts prompting framework.
- Fine-tuning SLMs using a synthesized dataset derived from LLM-generated responses.
- Evaluating the performance on the Game of 24 reasoning benchmark.
Key results:
- One-Step ToT significantly improves reasoning performance over Chain-of-Thought (CoT) prompting.
- The fine-tuned SLM achieves competitive accuracy with vastly more efficient resource utilization compared to LLMs.