From 0b1471c28e2233e2541f28b3d63f8e2933b1c986 Mon Sep 17 00:00:00 2001 From: youkaichao Date: Mon, 6 Jan 2025 21:20:36 +0800 Subject: [PATCH] add doc Signed-off-by: youkaichao --- docs/source/models/adding_model.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/source/models/adding_model.md b/docs/source/models/adding_model.md index 02537fba020c4..e54ef8697aaf6 100644 --- a/docs/source/models/adding_model.md +++ b/docs/source/models/adding_model.md @@ -153,3 +153,16 @@ Read more about that [here](#enabling-multimodal-inputs). ```{note} Although you can directly put these code snippets in your script using `vllm.LLM`, the recommended way is to place these snippets in a vLLM plugin. This ensures compatibility with various vLLM features like distributed inference and the API server. ``` + +## Frequently Asked Questions + +### How to support models with interleaving sliding windows? + +For models with interleaving sliding windows (e.g. `google/gemma-2-2b-it` and `mistralai/Ministral-8B-Instruct-2410`), the scheduler will treat the model as a full-attention model, i.e., kv-cache of all tokens will not be dropped. This is to make sure prefix caching works with these models. Sliding window only appears as a parameter to the attention kernel computation. + +To support a model with interleaving sliding windows, we need to take care of the following details: + +- Make sure [this line](https://github.com/vllm-project/vllm/blob/996357e4808ca5eab97d4c97c7d25b3073f46aab/vllm/config.py#L308) evaluates `has_interleaved_attention` to `True` for this model, and set `self.hf_text_config.interleaved_sliding_window` to the format of interleaving sliding windows the model can understand. Then, `self.hf_text_config.sliding_window` will be deleted, and the model will be treated as a full-attention model. +- In the modeling code, parse the correct sliding window value for every layer, and pass it to the attention layer's `per_layer_sliding_window` argument. For reference, check [this line](https://github.com/vllm-project/vllm/blob/996357e4808ca5eab97d4c97c7d25b3073f46aab/vllm/model_executor/models/llama.py#L171). + +With these two steps, interleave sliding windows should work with the model.