Skip to content

Commit

Permalink
Merge branch 'bitblas-intg' of https://github.com/LeiWang1999/vllm-bi…
Browse files Browse the repository at this point in the history
…tblas into bitblas-intg
  • Loading branch information
LeiWang1999 committed Dec 19, 2024
2 parents bcbad57 + 430ca44 commit 6cc9022
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions docs/source/quantization/bitblas.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ You can find pre-quantized models on https://huggingface.co/models?other=bitblas
And usually, these repositories have a quantize_config.json file that includes a quantization_config section.

Read bitblas format checkpoint.
--------------------------
---------------------------------

.. code-block:: python
Expand All @@ -30,7 +30,8 @@ Read bitblas format checkpoint.
llm = LLM(model=model_id, dtype=torch.bfloat16, trust_remote_code=True, quantization="bitblas")
Read gptq format checkpoint.
--------------------------
-------------------------------

.. code-block:: python
from vllm import LLM
Expand Down

0 comments on commit 6cc9022

Please sign in to comment.