Skip to content
This repository has been archived by the owner on Dec 1, 2024. It is now read-only.

Commit

Permalink
Update paper.md (#102)
Browse files Browse the repository at this point in the history
  • Loading branch information
shotarok authored Mar 21, 2023
1 parent 3834bb3 commit bbc9ea9
Showing 1 changed file with 12 additions and 1 deletion.
13 changes: 12 additions & 1 deletion docs/paper.md
Original file line number Diff line number Diff line change
@@ -1 +1,12 @@
# Coming soon
[High-throughput Generative Inference of Large Language Models with a Single GPU](https://arxiv.org/pdf/2303.06865)
=====================================================================================

Authors:[Ying Sheng](https://arxiv.org/search/cs?searchtype=author&query=Sheng%2C+Y), [Lianmin Zheng](https://arxiv.org/search/cs?searchtype=author&query=Zheng%2C+L), [Binhang Yuan](https://arxiv.org/search/cs?searchtype=author&query=Yuan%2C+B), [Zhuohan Li](https://arxiv.org/search/cs?searchtype=author&query=Li%2C+Z), [Max Ryabinin](https://arxiv.org/search/cs?searchtype=author&query=Ryabinin%2C+M), [Daniel Y. Fu](https://arxiv.org/search/cs?searchtype=author&query=Fu%2C+D+Y), [Zhiqiang Xie](https://arxiv.org/search/cs?searchtype=author&query=Xie%2C+Z), [Beidi Chen](https://arxiv.org/search/cs?searchtype=author&query=Chen%2C+B), [Clark Barrett](https://arxiv.org/search/cs?searchtype=author&query=Barrett%2C+C), [Joseph E. Gonzalez](https://arxiv.org/search/cs?searchtype=author&query=Gonzalez%2C+J+E), [Percy Liang](https://arxiv.org/search/cs?searchtype=author&query=Liang%2C+P), [Christopher Ré](https://arxiv.org/search/cs?searchtype=author&query=R%C3%A9%2C+C), [Ion Stoica](https://arxiv.org/search/cs?searchtype=author&query=Stoica%2C+I), [Ce Zhang](https://arxiv.org/search/cs?searchtype=author&query=Zhang%2C+C)

> Abstract: The high computational and memory requirements of large language model (LLM) inference traditionally make it feasible only with multiple high-end accelerators. Motivated by the emerging demand for latency-insensitive tasks with batched processing, this paper initiates the study of high-throughput LLM inference using limited resources, such as a single commodity GPU. We present FlexGen, a high-throughput generation engine for running LLMs with limited GPU memory. FlexGen can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. Through a linear programming optimizer, it searches for efficient patterns to store and access tensors. FlexGen further compresses these weights and the attention cache to 4 bits with negligible accuracy loss. These techniques enable FlexGen to have a larger space of batch size choices and thus significantly increase maximum throughput. As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours. The code is available at [this https URL](https://github.com/FMInference/FlexGen)

Subjects:Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Performance (cs.PF)

Cite as: [arXiv:2303.06865](https://arxiv.org/abs/2303.06865) \[cs.LG\]
(or [arXiv:2303.06865v1](https://arxiv.org/abs/2303.06865v1) \[cs.LG\] for this version)

0 comments on commit bbc9ea9

Please sign in to comment.