](https://flagopen.baai.ac.cn/)
+
## Latest News
- **[2024/11]** Released [v0.6.0](https://github.com/FlagOpen/FlagScale/tree/release/v0.6.0):
- Introduced general multi-dimensional heterogeneous parallelism and CPU-based communication between different chips.
@@ -12,7 +14,9 @@
[FlagScale](https://github.com/FlagOpen/FlagScale.git) is a comprehensive toolkit designed to support the entire lifecycle of large models, developed with the backing of the Beijing Academy of Artificial Intelligence (BAAI). It builds on the strengths of several prominent open-source projects, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [vllm](https://github.com/vllm-project/vllm), to provide a robust, end-to-end solution for managing and scaling large models.
-The primary objective of FlagScale is to enable seamless scalability across diverse hardware architectures while maximizing computational resource efficiency and enhancing model performance. By offering essential components for model development, training, and deployment, FlagScale aims to serve as an indispensable toolkit for optimizing both the speed and effectiveness of large model workflows.
+The primary objective of FlagScale is to enable seamless scalability across diverse hardware architectures while maximizing computational resource efficiency and enhancing model performance. By offering essential components for model development, training, and deployment, FlagScale seeks to establish itself as an indispensable toolkit for optimizing both the speed and effectiveness of large model workflows.
+
+FlagScale is also a part of [FlagAI-Open](https://flagopen.baai.ac.cn/), an open-source initiative by BAAI that aims to foster an open-source ecosystem for AI technologies. It serves as a platform where developers, researchers, and AI enthusiasts can collaborate on various AI projects, contribute to the development of cutting-edge AI solutions, and share their work with the global community.
## Quick Start
@@ -43,13 +47,15 @@ We recommend using the latest release of [NGC's PyTorch container](https://catal
cd vllm
pip install .
- cd megatron-energon
- pip install .
+ pip install -e ./megatron-energon
+ cp -r megatron-energon/src/megatron/energon megatron/megatron
```
### Run a Task
-FlagScale provides a unified runner for various tasks, including training and inference. Simply specify the configuration file to run the task with a single command. The runner will automatically load the configurations and execute the task. The following example demonstrates how to run a distributed training task.
+FlagScale provides a unified runner for various tasks, including training，inference and serve. Simply specify the configuration file to run the task with a single command. The runner will automatically load the configurations and execute the task. The following example demonstrates how to run a distributed training task.
+
+#### Train
1. Start the distributed training job:
```sh
@@ -62,6 +68,18 @@ FlagScale provides a unified runner for various tasks, including training and in
python run.py --config-path ./examples/aquila/conf --config-name config action=stop
```
+#### Serve
+
+1. Start the server:
+ ```sh
+ python run.py --config-path ./examples/qwen/conf --config-name config_qwen2.5_7b action=run
+ ```
+2. Stop the server:
+ ```sh
+ python run.py --config-path ./examples/qwen/conf --config-name config_qwen2.5_7b action=stop
+ ```
+For more details, please refer to [Quick Start](./flagscale/serve/README.md).
+
## License
This project is licensed under the [Apache License (Version 2.0)](https://github.com/FlagOpen/FlagScale/blob/main/LICENSE). This project also contains other third-party components under other open-source licenses. See the [LICENSE](https://github.com/FlagOpen/FlagScale/blob/main/LICENSE) file for more information.
diff --git a/examples/llama/conf/train/train_llama2_7b_tp_hetero.yaml b/examples/llama/conf/train/train_llama2_7b_tp_hetero.yaml
deleted file mode 100644
index 90995e2d9..000000000
--- a/examples/llama/conf/train/train_llama2_7b_tp_hetero.yaml
+++ /dev/null
@@ -1,67 +0,0 @@
-system:
- tensor_model_parallel_size: 4
- pipeline_model_parallel_size: 3
- disable_bias_linear: True
- use_flash_attn: True
- sequence_parallel: True
- use_distributed_optimizer: True
- hetero_mode: pp
- hetero_device_types: A100
- hetero_current_device_type: A100
- hetero_pipeline_stages: [3,16,8,8]
- process_meshes: [4,1,1,2,1,2]
- precision:
- bf16: True
- initial_loss_scale: 16384
- min_loss_scale: 1.0
- logging:
- log_interval: 1
- checkpoint:
- save_interval: 100
-
-model:
- use_mcore_models: True
- transformer_impl: transformer_engine
- num_layers: 32
- hidden_size: 4096
- ffn_hidden_size: 11008
- num_attention_heads: 32
- seq_length: 4096
- group_query_attention: False
- num_query_groups: 8
- max_position_embeddings: 4096
- norm_epsilon: 1e-5
- use_rotary_position_embeddings: True
- no_position_embedding: True
- swiglu: True
- normalization: RMSNorm
- untie_embeddings_and_output_weights: True
- init_method_std: 0.02
- attention_dropout: 0.0
- hidden_dropout: 0.0
- weight_decay: 0.1
- clip_grad: 1.0
- train_iters: 30
- eval_iters: 0
- eval_interval: 2000
- micro_batch_size: 1
- global_batch_size: 32
-
- optimizer:
- weight_decay: 1e-2
- adam_beta1: 0.9
- adam_beta2: 0.95
- lr_scheduler:
- lr: 0.00015
- min_lr: 1.0e-5
- lr_warmup_fraction: .01
- lr_decay_iters: 1
- lr_decay_style: cosine
-
-data:
- data_path: ${data_path:??}
- split: 1
- tokenizer:
- tokenizer_type: Llama2Tokenizer
- tokenizer_model: examples/llama/tokenizer.model
- vocab_size: 32000
diff --git a/examples/llama/conf/train/train_llama3_8b_hetero.yaml b/examples/llama/conf/train/train_llama3_8b_hetero.yaml
new file mode 100644
index 000000000..c73be22c8
--- /dev/null
+++ b/examples/llama/conf/train/train_llama3_8b_hetero.yaml
@@ -0,0 +1,100 @@
+system:
+ tensor_model_parallel_size: 4
+ pipeline_model_parallel_size: 2
+ disable_bias_linear: True
+ use_flash_attn: True
+ sequence_parallel: True
+ use_distributed_optimizer: True
+ precision:
+ bf16: True
+ attention_softmax_in_fp32: true
+ accumulate_allreduce_grads_in_fp32: true
+ logging:
+ log_interval: 1
+ tensorboard_log_interval: 1
+ wandb_project: "train-llama3-8B"
+ wandb_exp_name: "train-test-8B"
+ checkpoint:
+ load: outputs_llama3/checkpoint_mc
+ save_interval: 10
+ finetune: True
+ ckpt_format: "torch"
+
+model:
+ use_mcore_models: True
+ transformer_impl: transformer_engine
+ num_layers: 32
+ hidden_size: 4096
+ ffn_hidden_size: 14336
+ num_attention_heads: 32
+ seq_length: 4096
+ group_query_attention: True
+ num_query_groups: 8
+ max_position_embeddings: 8192
+ norm_epsilon: 1e-5
+ use_rotary_position_embeddings: True
+ no_position_embedding: True
+ swiglu: True
+ normalization: RMSNorm
+ rotary_interleaved_patch: False
+ position_embedding_type: rope
+ rotary_base: 500000
+ untie_embeddings_and_output_weights: True
+ init_method_std: 0.02
+ attention_dropout: 0.0
+ hidden_dropout: 0.0
+ clip_grad: 1.0
+ train_samples: 200000
+ eval_iters: 100
+ eval_interval: 1000
+ micro_batch_size: 1
+ global_batch_size: 16
+
+ hetero:
+ enable_hetero: True
+ hetero_use_cpu_communication: True
+ # mesh format [tp1,cp1,ep1,dp1,pp1,(tp2,cp2...)]
+
+ # 2 mesh, diff tp dp pp
+ hetero_pipeline_layer_split: [18, 14]
+ hetero_process_meshes: [2, 1, 1, 4, 1, 4, 1, 1, 2, 1]
+ hetero_device_types: ["A800", "A100"]
+
+ standalone_embedding_stage: False
+ hetero_current_device_type: "A800"
+
+ # recompute:
+ # recompute_granularity: "full"
+ # recompute_method: "uniform"
+ # recompute_num_layers: 1
+
+ # ## pp 2 stages and num_micro_batches 4
+ # recompute_granularity_per_stage_micro_batch:
+ # - [1, 3, 0, 1, 0]
+ # - [1, 3, 1, 1, 1]
+ # recompute_method_per_stage_micro_batch:
+ # - [1, 3, 0, 1, 0]
+ # - [1, 3, 0, 1, 0]
+ # recompute_num_layers_per_stage_micro_batch:
+ # - [1, 3, 2, 1, 2]
+ # - [1, 3, 1, 1, 1]
+
+
+ optimizer:
+ weight_decay: 1e-2
+ adam_beta1: 0.9
+ adam_beta2: 0.95
+ lr_scheduler:
+ lr: 1.0e-5
+ min_lr: 1.0e-6
+ lr_warmup_fraction: .1
+ lr_decay_style: cosine
+
+data:
+ data_path: examples/llama/pile-openwebtext_text_document/pile-openwebtext_text_document
+ split: 1
+ tokenizer:
+ tokenizer_type: Llama3TokenizerFS
+ tokenizer_path: meta-llama3/Meta-Llama-3-8B
+ vocab_size: 128256
+ make_vocab_size_divisible_by: 64
diff --git a/examples/llava_onevision/conf/train/train_llava_onevision_1.5b.yaml b/examples/llava_onevision/conf/train/train_llava_onevision_1.5b.yaml
index 142e59a8f..c9b44e0bd 100644
--- a/examples/llava_onevision/conf/train/train_llava_onevision_1.5b.yaml
+++ b/examples/llava_onevision/conf/train/train_llava_onevision_1.5b.yaml
@@ -44,6 +44,11 @@ model:
hidden_dropout: 0.0
clip_grad: 1.0
train_iters: 10
+ profile: False
+ profile-step-start: 10
+ profile-step-end: 20
+ profile_ranks: 7
+ use_pytorch_profiler: True
eval_iters: 0
micro_batch_size: 2
global_batch_size: 512
diff --git a/vllm/vllm/v1/tokenizer/__init__.py b/examples/qwen/__init__.py
similarity index 100%
rename from vllm/vllm/v1/tokenizer/__init__.py
rename to examples/qwen/__init__.py
diff --git a/examples/qwen/conf/config_qwen2.5_1.5b.yaml b/examples/qwen/conf/config_qwen2.5_1.5b.yaml
new file mode 100644
index 000000000..c3ab52d9f
--- /dev/null
+++ b/examples/qwen/conf/config_qwen2.5_1.5b.yaml
@@ -0,0 +1,30 @@
+defaults:
+ - _self_
+ - train: train_qwen_2.5_1.5b
+ # - train: train_mixtral_1.8b
+
+experiment:
+ exp_name: train_qwen_2.5_1.5b
+ exp_dir: ./outputs # outputs ## log、checkpoints output path
+ task:
+ type: train
+ backend: megatron
+ entrypoint: ./flagscale/train/train_aquila.py
+ runner:
+ backend: torchrun
+ nnodes: 2
+ nproc_per_node: 8
+ hostfile: torchrun # Please replace with your actual hostfile path
+ envs:
+ CUDA_VISIBLE_DEVICES: "0,1,2,3,4,5,6,7"
+ CUDA_DEVICE_MAX_CONNECTIONS: 1
+ NCCL_SOCKET_IFNAME: eth0
+ NCCL_IB_DISABLE: 0
+ NCCL_IB_CUDA_SUPPORT: 1
+ NCCL_IB_GID_INDEX: 0
+ NCCL_DEBUG: INFO
+ OMP_NUM_THREADS: 4
+ GLOO_SOCKET_IFNAME: eth0
+ NCCL_IB_HCA: mlx5_2,mlx5_5
+
+action: run
diff --git a/examples/qwen/conf/config_qwen2.5_72b_tp.yaml b/examples/qwen/conf/config_qwen2.5_72b_tp.yaml
new file mode 100644
index 000000000..a68a1e752
--- /dev/null
+++ b/examples/qwen/conf/config_qwen2.5_72b_tp.yaml
@@ -0,0 +1,22 @@
+defaults:
+ - _self_
+ - serve: serve_qwen2.5_72b
+
+experiment:
+ exp_name: qwen2.5_72b
+ exp_dir: outputs/${experiment.exp_name}
+ task:
+ type: serve
+ backend: vllm
+ entrypoint: null
+ runner:
+ hostfile: null
+ envs:
+ CUDA_VISIBLE_DEVICES: 0,1,2,3
+ CUDA_DEVICE_MAX_CONNECTIONS: 1
+
+action: run
+
+hydra:
+ run:
+ dir: ${experiment.exp_dir}/hydra
diff --git a/examples/qwen/conf/config_qwen2.5_7b.yaml b/examples/qwen/conf/config_qwen2.5_7b.yaml
new file mode 100644
index 000000000..48609ecd3
--- /dev/null
+++ b/examples/qwen/conf/config_qwen2.5_7b.yaml
@@ -0,0 +1,22 @@
+defaults:
+ - _self_
+ - serve: serve_qwen2.5_7b
+
+experiment:
+ exp_name: qwen2.5_7b
+ exp_dir: outputs/${experiment.exp_name}
+ task:
+ type: serve
+ backend: vllm
+ entrypoint: null
+ runner:
+ hostfile: null
+ envs:
+ CUDA_VISIBLE_DEVICES: 0
+ CUDA_DEVICE_MAX_CONNECTIONS: 1
+
+action: run
+
+hydra:
+ run:
+ dir: ${experiment.exp_dir}/hydra
diff --git a/examples/qwen/conf/config_ssh_qwen2.5_7b.yaml b/examples/qwen/conf/config_ssh_qwen2.5_7b.yaml
new file mode 100644
index 000000000..3a94895e4
--- /dev/null
+++ b/examples/qwen/conf/config_ssh_qwen2.5_7b.yaml
@@ -0,0 +1,25 @@
+defaults:
+ - _self_
+ - serve: serve_qwen2.5_7b
+
+experiment:
+ exp_name: qwen2.5_7b
+ exp_dir: outputs/${experiment.exp_name}
+ task:
+ type: serve
+ backend: vllm
+ entrypoint: null
+ runner:
+ hostfile: /path/to/hostfile # type: {remote ip} slots={gpu num} type={gpu type} (like: x.x.x.x slots=8 type=A100)
+ ssh_port: 22 # replace with your ssh port
+ envs:
+ CUDA_VISIBLE_DEVICES: 0
+ CUDA_DEVICE_MAX_CONNECTIONS: 1
+ cmds:
+ before_start: source /root/miniconda3/bin/activate flagscale
+
+action: run
+
+hydra:
+ run:
+ dir: ${experiment.exp_dir}/hydra
diff --git a/examples/qwen/conf/serve/serve_qwen2.5_72b.yaml b/examples/qwen/conf/serve/serve_qwen2.5_72b.yaml
new file mode 100644
index 000000000..86d05e037
--- /dev/null
+++ b/examples/qwen/conf/serve/serve_qwen2.5_72b.yaml
@@ -0,0 +1,17 @@
+model_args:
+ vllm_model:
+ model-tag: /models/Qwen2.5-72B-Instruct
+ tensor-parallel-size: 4
+ gpu-memory-utilization: 0.9
+ max-model-len: 32768
+ max-num-seqs: 256
+ port: 4567
+ action-args:
+ - trust-remote-code
+ - enable-chunked-prefill
+
+deploy:
+ command-line-mode: true
+ models:
+ vllm_model:
+ num_gpus: 4
diff --git a/examples/qwen/conf/serve/serve_qwen2.5_7b.yaml b/examples/qwen/conf/serve/serve_qwen2.5_7b.yaml
new file mode 100644
index 000000000..0faa78b42
--- /dev/null
+++ b/examples/qwen/conf/serve/serve_qwen2.5_7b.yaml
@@ -0,0 +1,17 @@
+model_args:
+ vllm_model:
+ model-tag: /models/Qwen2.5-7B-Instruct
+ tensor-parallel-size: 1
+ gpu-memory-utilization: 0.9
+ max-model-len: 32768
+ max-num-seqs: 256
+ port: 4567
+ action-args:
+ - trust-remote-code
+ - enable-chunked-prefill
+
+deploy:
+ command-line-mode: true
+ models:
+ vllm_model:
+ num_gpus: 1
diff --git a/examples/qwen/conf/train/qwen_2.5_1.5b.yaml b/examples/qwen/conf/train/qwen_2.5_1.5b.yaml
new file mode 100644
index 000000000..2ea5368fb
--- /dev/null
+++ b/examples/qwen/conf/train/qwen_2.5_1.5b.yaml
@@ -0,0 +1,80 @@
+system:
+ tensor_model_parallel_size: 1
+ pipeline_model_parallel_size: 1
+ make_vocab_size_divisible_by: 128
+ disable_bias_linear: True
+ sequence_parallel: True
+ use_flash_attn: True
+ use_distributed_optimizer: True
+ distributed-timeout-minutes: 60
+ precision:
+ bf16: True
+ attention_softmax_in_fp32: True
+ accumulate_allreduce_grads_in_fp32: True
+ logging:
+ log_interval: 1
+ tensorboard_log_interval: 1
+ wandb_project: "train-qwen2.5-1.5B"
+ wandb_exp_name: "train-qwen2.5-1.5B"
+ checkpoint:
+ load: ${megatron_model__path:}
+ # If you want to train the model, you need to comment out ckpt_format, ckpt_convert_format, ckpt_convert_save, which are used for converting ckpt.
+ ckpt_format: torch_dist # ${experiment.ckpt_format}
+ ckpt_convert_format: torch # ${experiment.ckpt_convert_format}
+ ckpt_convert_save: ${experiment.ckpt_convert_save}
+ save_interval: 5000000
+ rampup_save_interval: 50000
+
+model:
+ use_mcore_models: true
+ num_layers: 28
+ hidden_size: 1536
+ num_attention_heads: 12
+ num_query_groups: 2
+ group_query_attention: True
+ ffn_hidden_size: 8960
+ seq_length: 4096
+ max_position_embeddings: 4096
+ norm_epsilon: 1e-6
+ norm_init_weight: 0.02
+ use_rotary_position_embeddings: true
+ rotary_base: 1000000.0
+ no_position_embedding: true
+ reset_position_ids: true
+ add_qkv_bias: true
+ reset_attention_mask: true
+ swiglu: true
+ normalization: RMSNorm
+ untie_embeddings_and_output_weights: false
+ init_method_std: 0.02
+ attention_dropout: 0.0
+ hidden_dropout: 0.0
+ weight_decay: 0.0
+ clip_grad: 1.0
+ train_samples: 1478125
+ eval_iters: 0
+ eval_interval: 2000000
+ micro_batch_size: 1
+ global_batch_size: 512
+ finetune: true
+ transformer_impl: transformer_engine
+ seed: 42
+ #data_searching_range: [1156,1274]
+ optimizer:
+ weight_decay: 0.0
+ adam_beta1: 0.9
+ adam_beta2: 0.95
+ lr_scheduler:
+ lr: 1e-5
+ min_lr: 0
+ lr_warmup_samples: 21120
+ lr_decay_style: cosine
+
+data:
+ data_path: ${data_path:??}
+ split: 1
+ apply_sft_dataset_separated_loss_mask_if_existed: true
+ tokenizer:
+ tokenizer_type: HFTokenizerFS
+ tokenizer_path: ${HF_model_path:??}
+ vocab_size: 151665
diff --git a/examples/qwen/utils/__init__.py b/examples/qwen/utils/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/examples/qwen/utils/convo_dataset.py b/examples/qwen/utils/convo_dataset.py
new file mode 100644
index 000000000..a9234b7d2
--- /dev/null
+++ b/examples/qwen/utils/convo_dataset.py
@@ -0,0 +1,237 @@
+"""GPT style dataset."""
+
+import copy
+import hashlib
+import os
+import time
+
+import numpy as np
+import torch
+
+from megatron import print_rank_0
+from megatron.core import mpu
+from megatron.data.data_samplers import RandomSeedDataset
+
+class ConversationDatasetCPT(torch.utils.data.Dataset):
+ def __init__(self, conversations, tokenizer, maxlen, seed, num_samples, role_sep="\n\n"):
+ super(ConversationDatasetCPT, self).__init__()
+ self.conversations = conversations
+ self.tokenizer = tokenizer
+ self.maxlen = maxlen+1
+ self.seed = seed
+ self.num_samples = num_samples
+
+ ## TODO convo template
+ self.sep = role_sep
+
+ # rng state
+ np_rng = np.random.RandomState(seed=seed)
+ np_rng.shuffle(self.conversations)
+
+ def __getitem__(self, i):
+ source = self.conversations[i]
+
+ instruction = source['instruction']
+ conversations = source['conversations']
+
+ BOS_TOKEN = self.tokenizer.cls
+ EOS_TOKEN = self.tokenizer.eod
+ example = [BOS_TOKEN]
+
+ # instruction
+ instruction = self.tokenizer.tokenize(f"{instruction}")
+ example += instruction
+
+ labels = [-100] * len(example)
+
+ for conversation in conversations:
+ role = conversation['from']
+ content = conversation['value']
+ content += self.sep
+
+ content = self.tokenizer.tokenize(f"{content}")
+
+ example += content
+ if role == 'gpt':
+ role_labels = copy.deepcopy(content)
+ else:
+ # masking
+ role_labels = [-100] * len(content)
+ labels += role_labels
+
+ example.append(EOS_TOKEN)
+ labels.append(EOS_TOKEN)
+
+ # maxlen
+ example = example[:self.maxlen]
+ labels = labels[:self.maxlen]
+
+ # padding
+ delta = self.maxlen - len(example)
+ if delta > 0:
+ example.extend([self.tokenizer.pad]*delta)
+ labels.extend([-100]*delta)
+
+ output = {
+ "tokens": np.array(example, dtype=np.int64),
+ "labels": np.array(labels, dtype=np.int64),
+ }
+ return output
+
+ def __len__(self):
+ return len(self.conversations)
+
+
+class ConversationDatasetV2(torch.utils.data.Dataset):
+ def __init__(self, conversations, tokenizer, maxlen, seed, num_samples):
+ super(ConversationDatasetV2, self).__init__()
+ self.conversations = conversations
+ self.tokenizer = tokenizer
+ self.maxlen = maxlen+1
+ self.seed = seed
+ self.num_samples = num_samples
+
+ # rng state
+ np_rng = np.random.RandomState(seed=seed)
+ np_rng.shuffle(self.conversations)
+
+
+ def __getitem__(self, i):
+ from examples.aquila.utils.convo_prompt import _add_speaker_and_signal
+ from examples.aquila.utils.convo_prompt import header
+
+ #source = self.conversations[self.sample_idx[i]]
+ source = self.conversations[i]
+ _add_speaker_and_signal(source)
+
+ source["chat_desc"] = header
+ chat_desc = source['chat_desc']
+ instruction = source['instruction']
+ conversations = source['conversations']
+
+ BOS_TOKEN = self.tokenizer.cls
+ EOS_TOKEN = self.tokenizer.eod
+ example = [BOS_TOKEN]
+
+ # chat_desc
+ example += self.tokenizer.tokenize(f"{chat_desc}")
+
+ # instruction
+ instruction = self.tokenizer.tokenize(f"{instruction}")
+ example += instruction
+
+ labels = copy.deepcopy(example)
+ # add zero-out
+ #labels = [-100] * len(example)
+
+ for conversation in conversations:
+ role = conversation['from']
+ content = conversation['value']
+ content = self.tokenizer.tokenize(f"{content}")
+ example += content
+ if role == 'gpt':
+ role_labels = copy.deepcopy(content)
+ else:
+ # masking
+ role_labels = [-100] * len(content)
+ labels += role_labels
+
+ example.append(EOS_TOKEN)
+ labels.append(EOS_TOKEN)
+
+ # maxlen
+ example = example[:self.maxlen]
+ labels = labels[:self.maxlen]
+
+ # padding
+ delta = self.maxlen - len(example)
+ if delta > 0:
+ example.extend([self.tokenizer.pad]*delta)
+ labels.extend([-100]*delta)
+
+ output = {
+ "tokens": np.array(example, dtype=np.int64),
+ "labels": np.array(labels, dtype=np.int64),
+ }
+ return output
+
+ def __len__(self):
+ #return len(self.sample_idx)
+ return len(self.conversations)
+
+
+def build_train_valid_test_datasets(train_valid_test_num_samples,
+ seq_length, seed, tokenizer,
+ train_data_prefix,
+ valid_data_prefix,
+ test_data_prefix=None,
+ finetune_dataset_type=None):
+ """Build train, valid, and test datasets."""
+ suppored_dataset_types = dict(CPT=ConversationDatasetCPT)
+ dataset_cls = ConversationDatasetV2
+ if finetune_dataset_type in suppored_dataset_types:
+ dataset_cls = suppored_dataset_types[finetune_dataset_type]
+
+ def read_file(jsonl_file):
+ import jsonlines
+ conversations = []
+ with jsonlines.open(jsonl_file) as reader:
+ for line in reader:
+ conversations.append(line)
+ return conversations
+
+ train_dataset, valid_dataset, test_dataset = None, None, None
+ # Single dataset.
+ if train_data_prefix is not None:
+ train_conversations = read_file(train_data_prefix[0])
+ train_dataset = dataset_cls(
+ train_conversations,
+ tokenizer=tokenizer,
+ maxlen=seq_length,
+ seed=seed,
+ num_samples=train_valid_test_num_samples[0])
+ train_dataset = RandomSeedDataset(train_dataset)
+
+ if valid_data_prefix is not None:
+ valid_conversations = read_file(valid_data_prefix[0])
+ valid_dataset = dataset_cls(
+ valid_conversations,
+ tokenizer=tokenizer,
+ maxlen=seq_length,
+ seed=seed,
+ num_samples=train_valid_test_num_samples[1])
+ valid_dataset = RandomSeedDataset(valid_dataset)
+
+ if test_data_prefix is not None:
+ test_conversations = read_file(test_data_prefix[0])
+ test_dataset = dataset_cls(
+ test_conversations,
+ tokenizer=tokenizer,
+ maxlen=seq_length,
+ seed=seed,
+ num_samples=train_valid_test_num_samples[2])
+ test_dataset = RandomSeedDataset(test_dataset)
+
+ return (train_dataset, valid_dataset, test_dataset)
+
+if __name__ == "__main__":
+ train_valid_test_num_samples = [12000,2000,0]
+ seq_length = 2048
+ seed = 1234
+ from megatron.tokenizer.tokenizer import _AquilaTokenizer
+ tokenizer = _AquilaTokenizer(
+ '../examples/aquila/tokenizer/vocab.json',
+ '../examples/aquila/tokenizer/merges.txt')
+ print(f"{dir(tokenizer)}")
+ train_data_prefix = ['path/to/train/set']
+ valid_data_prefix = ['path/to/valid/set']
+ train_dataset, valid_dataset, test_dataset = build_train_valid_test_datasets(
+ train_valid_test_num_samples,
+ seq_length, seed, tokenizer,
+ train_data_prefix,
+ valid_data_prefix,
+ test_data_prefix=None)
+ for idx, sample in enumerate(train_dataset):
+ print(f"idx={idx} sample={type(sample['labels'])}")
+ break
+
diff --git a/examples/qwen/utils/convo_prompt.py b/examples/qwen/utils/convo_prompt.py
new file mode 100644
index 000000000..e0e947282
--- /dev/null
+++ b/examples/qwen/utils/convo_prompt.py
@@ -0,0 +1,166 @@
+import dataclasses
+from enum import auto, Enum
+from typing import List, Tuple, Any
+
+
+class SeparatorStyle(Enum):
+ """Different separator style."""
+ SINGLE = auto()
+ TWO = auto()
+
+
+@dataclasses.dataclass
+class Conversation:
+ """A class that keeps all conversation history."""
+ system: str
+ instruction: str
+ roles: List[str]
+ messages: List[List[str]]
+ offset: int
+ sep_style: SeparatorStyle = SeparatorStyle.SINGLE
+ sep: str = "###"
+ sep2: str = None
+
+ skip_next: bool = False
+ conv_id: Any = None
+
+ def get_prompt(self):
+ if self.sep_style == SeparatorStyle.SINGLE:
+ ret = self.system + self.sep
+ if self.instruction is not None and len(self.instruction) > 0:
+ ret += self.roles[2] + ": " + self.instruction + self.sep
+ for role, message in self.messages:
+ if message:
+ ret += role + ": " + message + self.sep
+ else:
+ ret += role + ":"
+ return ret
+ elif self.sep_style == SeparatorStyle.TWO:
+ seps = [self.sep, self.sep2]
+ ret = self.system + seps[0]
+ if self.instruction is not None and len(self.instruction) > 0:
+ ret += self.roles[2] + ": " + self.instruction + self.sep
+ for i, (role, message) in enumerate(self.messages):
+ if message:
+ ret += role + ": " + message + seps[i % 2]
+ else:
+ ret += role + ":"
+ return ret
+ else:
+ raise ValueError(f"Invalid style: {self.sep_style}")
+
+ def append_message(self, role, message):
+ self.messages.append([role, message])
+
+ def to_gradio_chatbot(self):
+ ret = []
+ for i, (role, msg) in enumerate(self.messages[self.offset:]):
+ if i % 2 == 0:
+ ret.append([msg, None])
+ else:
+ ret[-1][-1] = msg
+ return ret
+
+ def copy(self):
+ return Conversation(
+ system=self.system,
+ instruction=self.instruction,
+ roles=self.roles,
+ messages=[[x, y] for x, y in self.messages],
+ offset=self.offset,
+ sep_style=self.sep_style,
+ sep=self.sep,
+ sep2=self.sep2,
+ conv_id=self.conv_id)
+
+ def dict(self):
+ return {
+ "system": self.system,
+ "instruction": self.instruction,
+ "roles": self.roles,
+ "messages": self.messages,
+ "offset": self.offset,
+ "sep": self.sep,
+ "sep2": self.sep2,
+ "conv_id": self.conv_id,
+ }
+
+
+conv_v1 = Conversation(
+ system="A chat between a curious human and an artificial intelligence assistant. "
+ "The assistant gives helpful, detailed, and polite answers to the human's questions.",
+ instruction="",
+ roles=("Human", "Assistant", "System"),
+ messages=(),
+ offset=0,
+ sep_style=SeparatorStyle.SINGLE,
+ sep="###",
+)
+
+conv_v1_2 = Conversation(
+ system="A chat between a curious human and an artificial intelligence assistant. "
+ "The assistant gives helpful, detailed, and polite answers to the human's questions.",
+ instruction="",
+ roles=("Human", "Assistant", "System"),
+ messages=(),
+ offset=0,
+ sep_style=SeparatorStyle.SINGLE,
+ sep="###",
+)
+
+conv_bair_v1 = Conversation(
+ system="BEGINNING OF CONVERSATION:",
+ instruction="",
+ roles=("USER", "GPT", "System"),
+ messages=(),
+ offset=0,
+ sep_style=SeparatorStyle.TWO,
+ sep=" ",
+ sep2="",
+)
+
+
+default_conversation = conv_v1_2
+conv_templates = {
+ "v1": conv_v1_2,
+ "bair_v1": conv_bair_v1,
+}
+
+# utils
+"""Add speaker and start/end signal on each round."""
+BEGIN_SIGNAL = "### "
+END_SIGNAL = "\n"
+header = f"{default_conversation.system}\n\n"
+unknown_role = "unknown" # use default unknown role
+roles = {
+ "human": default_conversation.roles[0], # human role
+ "gpt": default_conversation.roles[1], # gpt role
+}
+
+def _add_speaker_and_signal(source, get_conversation=True):
+ conversation = header
+ if "instruction" in source and source["instruction"] is not None and len(source["instruction"]) > 0:
+ source["instruction"] = (
+ BEGIN_SIGNAL
+ + conversation_lib.default_conversation.roles[2]
+ + ": "
+ + source["instruction"]
+ + END_SIGNAL
+ )
+ if get_conversation:
+ conversation += source["instruction"]
+ for sentence in source["conversations"]:
+ sentence_from = sentence["from"].lower()
+ sentence["value"] = (
+ BEGIN_SIGNAL
+ + roles.get(sentence_from, unknown_role)
+ + ": "
+ + sentence["value"]
+ + END_SIGNAL
+ )
+ if get_conversation:
+ conversation += sentence["value"]
+ return conversation
+
+if __name__ == "__main__":
+ print(default_conversation.get_prompt())
diff --git a/examples/qwen/utils/cyg_conversation.py b/examples/qwen/utils/cyg_conversation.py
new file mode 100644
index 000000000..6bb9c3a0f
--- /dev/null
+++ b/examples/qwen/utils/cyg_conversation.py
@@ -0,0 +1,157 @@
+import dataclasses
+from enum import auto, Enum
+from typing import List, Tuple, Any
+
+
+class SeparatorStyle(Enum):
+ """Different separator style."""
+ SINGLE = auto()
+ TWO = auto()
+
+
+@dataclasses.dataclass
+class Conversation:
+ """A class that keeps all conversation history."""
+ system: str
+ instruction: str
+ roles: List[str]
+ messages: List[List[str]]
+ offset: int
+ sep_style: SeparatorStyle = SeparatorStyle.SINGLE
+ sep: str = "###"
+ sep2: str = None
+
+ skip_next: bool = False
+ conv_id: Any = None
+
+ def get_prompt(self):
+ if self.sep_style == SeparatorStyle.SINGLE:
+ ret = self.system + self.sep
+ if self.instruction is not None and len(self.instruction) > 0:
+ ret += self.roles[2] + ": " + self.instruction + self.sep
+ for role, message in self.messages:
+ if message:
+ ret += role + ": " + message + self.sep
+ else:
+ ret += role + ":"
+ return ret
+ elif self.sep_style == SeparatorStyle.TWO:
+ seps = [self.sep, self.sep2]
+ ret = self.system + seps[0]
+ if self.instruction is not None and len(self.instruction) > 0:
+ ret += self.roles[2] + ": " + self.instruction + self.sep
+ for i, (role, message) in enumerate(self.messages):
+ if message:
+ ret += role + ": " + message + seps[i % 2]
+ else:
+ ret += role + ":"
+ return ret
+ else:
+ raise ValueError(f"Invalid style: {self.sep_style}")
+
+ def append_message(self, role, message):
+ self.messages.append([role, message])
+
+ def to_gradio_chatbot(self):
+ ret = []
+ for i, (role, msg) in enumerate(self.messages[self.offset:]):
+ if i % 2 == 0:
+ ret.append([msg, None])
+ else:
+ ret[-1][-1] = msg
+ return ret
+
+ def copy(self):
+ return Conversation(
+ system=self.system,
+ instruction=self.instruction,
+ roles=self.roles,
+ messages=[[x, y] for x, y in self.messages],
+ offset=self.offset,
+ sep_style=self.sep_style,
+ sep=self.sep,
+ sep2=self.sep2,
+ conv_id=self.conv_id)
+
+ def dict(self):
+ return {
+ "system": self.system,
+ "instruction": self.instruction,
+ "roles": self.roles,
+ "messages": self.messages,
+ "offset": self.offset,
+ "sep": self.sep,
+ "sep2": self.sep2,
+ "conv_id": self.conv_id,
+ }
+
+
+conv_v1 = Conversation(
+ system="A chat between a curious human and an artificial intelligence assistant. "
+ "The assistant gives helpful, detailed, and polite answers to the human's questions.",
+ instruction="",
+ roles=("Human", "Assistant", "System"),
+ messages=(),
+ offset=0,
+ sep_style=SeparatorStyle.SINGLE,
+ sep="###",
+)
+
+conv_v1_2 = Conversation(
+ system="A chat between a curious human and an artificial intelligence assistant. "
+ "The assistant gives helpful, detailed, and polite answers to the human's questions.",
+ instruction="",
+ roles=("Human", "Assistant", "System"),
+ messages=(),
+ offset=0,
+ sep_style=SeparatorStyle.SINGLE,
+ sep="###",
+)
+
+conv_bair_v1 = Conversation(
+ system="BEGINNING OF CONVERSATION:",
+ instruction="",
+ roles=("USER", "GPT", "System"),
+ messages=(),
+ offset=0,
+ sep_style=SeparatorStyle.TWO,
+ sep=" ",
+ sep2="",
+)
+
+
+default_conversation = conv_v1_2
+conv_templates = {
+ "v1": conv_v1_2,
+ "bair_v1": conv_bair_v1,
+}
+
+
+def covert_prompt_to_input_ids_with_history(text, history, tokenizer, max_token):
+ conv = default_conversation.copy()
+
+ conv.append_message(conv.roles[1], None)
+ conv.append_message(conv.roles[0], text)
+
+ example = tokenizer.encode_plus(f"{conv.get_prompt()} ", None, max_length=None)['input_ids']
+
+ while(len(history) > 0 and (len(example) < max_token)):
+ tmp = history.pop()
+ if tmp[0] == 'ASSISTANT':
+ conv.append_message(conv.roles[1], tmp[1])
+ else:
+ conv.append_message(conv.roles[0], tmp[1])
+ example = tokenizer.encode_plus(f"{conv.get_prompt()} ", None, max_length=None)['input_ids']
+
+ if len(example) >= max_token:
+ conv.messages.pop()
+ conv.messages = conv.messages[::-1]
+ print('model in:', conv.get_prompt())
+ example = tokenizer.encode_plus(f"{conv.get_prompt()} ", None, max_length=None)['input_ids']
+ #example = example[1:-1]
+
+ return example
+
+if __name__ == "__main__":
+ print(default_conversation.get_prompt())
+
diff --git a/flagopen.png b/flagopen.png
new file mode 100644
index 000000000..82ffee806
Binary files /dev/null and b/flagopen.png differ
diff --git a/flagscale/inference/core/block_manager.py b/flagscale/inference/core/block_manager.py
index 34a0d3d52..93770b8dc 100644
--- a/flagscale/inference/core/block_manager.py
+++ b/flagscale/inference/core/block_manager.py
@@ -16,9 +16,7 @@
SeqId = int
EncoderSeqId = str
-# --- FLAGSCALE MODIFICATION BEG ---
-NegativeSeqId = int
-# --- FLAGSCALE MODIFICATION END ---
+NegativeSeqId = int # --- FLAGSCALE MODIFICATION ---
class SelfAttnBlockSpaceManager(BlockSpaceManager):
@@ -103,12 +101,10 @@ def __init__(
self.block_tables: Dict[SeqId, BlockTable] = {}
self.cross_block_tables: Dict[EncoderSeqId, BlockTable] = {}
- # --- FLAGSCALE MODIFICATION BEG ---
- self.negative_block_tables: Dict[NegativeSeqId, BlockTable] = {}
- # --- FLAGSCALE MODIFICATION END ---
+ self.negative_block_tables: Dict[NegativeSeqId, BlockTable] = {} # --- FLAGSCALE MODIFICATION ---
self._computed_blocks_tracker = ComputedBlocksTracker(
- self.block_allocator)
+ self.block_allocator, self.block_size, self.enable_caching)
self._last_access_blocks_tracker = LastAccessBlocksTracker(
self.block_allocator)
@@ -187,7 +183,6 @@ def allocate(self, seq_group: SequenceGroup) -> None:
self.block_tables[seq.seq_id] = block_table
# Track seq
- self._computed_blocks_tracker.add_seq(seq.seq_id)
self._last_access_blocks_tracker.add_seq(seq.seq_id)
# Assign the block table for each sequence.
@@ -195,7 +190,6 @@ def allocate(self, seq_group: SequenceGroup) -> None:
self.block_tables[seq.seq_id] = block_table.fork()
# Track seq
- self._computed_blocks_tracker.add_seq(seq.seq_id)
self._last_access_blocks_tracker.add_seq(seq.seq_id)
# Allocate cross-attention block table for encoder sequence
@@ -380,11 +374,13 @@ def get_common_computed_block_ids(
"""
computed_seq_block_ids = []
for seq in seqs:
- computed_seq_block_ids.append(
- self._computed_blocks_tracker.
- get_cached_computed_blocks_and_update(
- seq.seq_id,
- self.block_tables[seq.seq_id].physical_block_ids))
+ all_blocks = self.block_tables[seq.seq_id].physical_block_ids
+ num_cached_tokens = (
+ self._computed_blocks_tracker.get_num_cached_tokens(seq))
+ assert num_cached_tokens % self.block_size == 0
+ num_cached_blocks = num_cached_tokens // self.block_size
+ computed_block_ids = all_blocks[:num_cached_blocks]
+ computed_seq_block_ids.append(computed_block_ids)
# NOTE(sang): This assumes seq_block_ids doesn't contain any None.
return self.block_allocator.get_common_computed_block_ids(
@@ -398,7 +394,6 @@ def fork(self, parent_seq: Sequence, child_seq: Sequence) -> None:
self.block_tables[child_seq.seq_id] = src_block_table.fork()
# Track child seq
- self._computed_blocks_tracker.add_seq(child_seq.seq_id)
self._last_access_blocks_tracker.add_seq(child_seq.seq_id)
def can_swap_in(self, seq_group: SequenceGroup,
@@ -459,7 +454,7 @@ def can_swap_out(self, seq_group: SequenceGroup) -> bool:
with num_lookahead_slots.
Args:
- seq_group (SequenceGroup): The sequence group to swap in.
+ seq_group (SequenceGroup): The sequence group to swap out.
num_lookahead_slots (int): Number of lookahead slots used in
speculative decoding, default to 0.
@@ -475,7 +470,7 @@ def swap_out(self, seq_group: SequenceGroup) -> List[Tuple[int, int]]:
swapping out the given sequence_group with num_lookahead_slots.
Args:
- sequence_group (SequenceGroup): The sequence group to swap in.
+ sequence_group (SequenceGroup): The sequence group to swap out.
Returns:
List[Tuple[int, int]]: The mapping of swapping block from
@@ -525,7 +520,7 @@ def _can_swap(self,
on to the 'device'.
Args:
- sequence_group (SequenceGroup): The sequence group to swap in.
+ sequence_group (SequenceGroup): The sequence group to swap in/out.
device (Device): device to swap the 'seq_group' on.
status (SequenceStatus): The status of sequence which is needed
for action. RUNNING for swap out and SWAPPED for swap in
@@ -569,3 +564,9 @@ def _can_swap(self,
return AllocStatus.OK
else:
return AllocStatus.LATER
+
+ def get_num_cached_tokens(self, seq: Sequence) -> int:
+ """Get the number of tokens in blocks that are already computed and
+ cached in the block manager for the sequence.
+ """
+ return self._computed_blocks_tracker.get_num_cached_tokens(seq)
diff --git a/flagscale/inference/core/data.py b/flagscale/inference/core/data.py
index 55b7240c0..5321d6a6e 100644
--- a/flagscale/inference/core/data.py
+++ b/flagscale/inference/core/data.py
@@ -1,11 +1,16 @@
# This file is modified from 'FlagScale/vllm/vllm/inputs/data.py'
-from typing import (TYPE_CHECKING, Any, Dict, Generic, Iterable, List,
+from dataclasses import dataclass
+from functools import cached_property
+from typing import (TYPE_CHECKING, Any, Dict, Generic, Iterable, List, Literal,
Optional, Tuple, Union, cast)
-from typing_extensions import NotRequired, TypedDict, TypeVar
+import torch
+from typing_extensions import NotRequired, TypedDict, TypeVar, assert_never
if TYPE_CHECKING:
- from vllm.multimodal import MultiModalDataDict
+ from vllm.multimodal import (MultiModalDataDict, MultiModalKwargs,
+ MultiModalPlaceholderDict)
+ from vllm.multimodal.inputs import MultiModalInputsV2
class TextPrompt(TypedDict):
@@ -40,15 +45,18 @@ class TokensPrompt(TypedDict):
prompt_token_ids: List[int]
"""A list of token IDs to pass to the model."""
+ token_type_ids: NotRequired[List[int]]
+ """A list of token type IDs to pass to the cross encoder model."""
+
multi_modal_data: NotRequired["MultiModalDataDict"]
"""
- Optional multi-modal data to pass to the model,
+ DEPRECATED: Optional multi-modal data to pass to the model,
if the model supports it.
"""
mm_processor_kwargs: NotRequired[Dict[str, Any]]
"""
- Optional multi-modal processor kwargs to be forwarded to the
+ DEPRECATED: Optional multi-modal processor kwargs to be forwarded to the
multimodal input mapper & processor. Note that if multiple modalities
have registered mappers etc for the model being considered, we attempt
to pass the mm_processor_kwargs to each of them.
@@ -133,21 +141,39 @@ class ExplicitEncoderDecoderPrompt(TypedDict, Generic[_T1_co, _T2_co]):
class TokenInputs(TypedDict):
"""Represents token-based inputs."""
+
+ type: Literal["token"]
+ """The type of inputs."""
+
prompt_token_ids: List[int]
"""The token IDs of the prompt."""
- prompt: NotRequired[Optional[str]]
+ token_type_ids: NotRequired[List[int]]
+ """The token type IDs of the prompt."""
+
+ prompt: NotRequired[str]
"""
The original prompt text corresponding to the token IDs, if available.
"""
- multi_modal_data: NotRequired[Optional["MultiModalDataDict"]]
+ multi_modal_data: NotRequired["MultiModalDataDict"]
"""
Optional multi-modal data to pass to the model,
if the model supports it.
"""
- mm_processor_kwargs: NotRequired[Optional[Dict[str, Any]]]
+ multi_modal_inputs: NotRequired["MultiModalKwargs"]
+ """
+ Optional multi-modal inputs to pass to the model,
+ if the model supports it.
+ """
+
+ multi_modal_placeholders: NotRequired["MultiModalPlaceholderDict"]
+ """
+ Placeholder ranges for the multi-modal data.
+ """
+
+ mm_processor_kwargs: NotRequired[Dict[str, Any]]
"""
Optional multi-modal processor kwargs to be forwarded to the
multimodal input mapper & processor. Note that if multiple modalities
@@ -163,8 +189,11 @@ class TokenInputs(TypedDict):
def token_inputs(
prompt_token_ids: List[int],
+ token_type_ids: Optional[List[int]] = None,
prompt: Optional[str] = None,
multi_modal_data: Optional["MultiModalDataDict"] = None,
+ multi_modal_inputs: Optional["MultiModalKwargs"] = None,
+ multi_modal_placeholders: Optional["MultiModalPlaceholderDict"] = None,
mm_processor_kwargs: Optional[Dict[str, Any]] = None,
# --- FLAGSCALE MODIFICATION BEG ---
negative_prompt_token_ids: Optional[List[int]] = None,
@@ -173,31 +202,31 @@ def token_inputs(
) -> TokenInputs:
"""Construct :class:`TokenInputs` from optional values."""
# --- FLAGSCALE MODIFICATION BEG ---
- inputs = TokenInputs(prompt_token_ids=prompt_token_ids,
+ inputs = TokenInputs(type="token", prompt_token_ids=prompt_token_ids,
negative_prompt_token_ids=negative_prompt_token_ids)
# --- FLAGSCALE MODIFICATION END ---
if prompt is not None:
inputs["prompt"] = prompt
- if multi_modal_data is not None:
- inputs["multi_modal_data"] = multi_modal_data
- if mm_processor_kwargs is not None:
- inputs["mm_processor_kwargs"] = mm_processor_kwargs
# --- FLAGSCALE MODIFICATION BEG ---
if negative_prompt is not None:
inputs["negative_prompt"] = negative_prompt
# --- FLAGSCALE MODIFICATION END ---
+ if token_type_ids is not None:
+ inputs["token_type_ids"] = token_type_ids
+ if multi_modal_data is not None:
+ inputs["multi_modal_data"] = multi_modal_data
+ if multi_modal_inputs is not None:
+ inputs["multi_modal_inputs"] = multi_modal_inputs
+ if multi_modal_placeholders is not None:
+ inputs["multi_modal_placeholders"] = multi_modal_placeholders
+ if mm_processor_kwargs is not None:
+ inputs["mm_processor_kwargs"] = mm_processor_kwargs
return inputs
-SingletonInputs = TokenInputs
-"""
-A processed :class:`SingletonPrompt` which can be passed to
-:class:`vllm.sequence.Sequence`.
-"""
-
-DecoderOnlyInputs = TokenInputs
+DecoderOnlyInputs = Union[TokenInputs, "MultiModalInputsV2"]
"""
The inputs in :class:`~vllm.LLMEngine` before they are
passed to the model executor.
@@ -205,28 +234,123 @@ def token_inputs(
"""
-class EncoderDecoderInputs(TokenInputs):
+class EncoderDecoderInputs(TypedDict):
"""
The inputs in :class:`~vllm.LLMEngine` before they are
passed to the model executor.
This specifies the required data for encoder-decoder models.
"""
- encoder_prompt_token_ids: List[int]
- """The token IDs of the encoder prompt."""
+ encoder: Union[TokenInputs, "MultiModalInputsV2"]
+ """The inputs for the encoder portion."""
+
+ decoder: Union[TokenInputs, "MultiModalInputsV2"]
+ """The inputs for the decoder portion."""
- encoder_prompt: NotRequired[Optional[str]]
- """
- The original encoder prompt text corresponding to the token IDs, if
- available.
- """
- encoder_multi_modal_data: NotRequired[Optional["MultiModalDataDict"]]
+SingletonInputs = Union[TokenInputs, "MultiModalInputsV2"]
+"""
+A processed :class:`SingletonPrompt` which can be passed to
+:class:`vllm.sequence.Sequence`.
+"""
+
+
+@dataclass
+class SingletonInputsAdapter:
"""
- Optional multi-modal data to pass to the encoder model,
- if the model supports it.
+ Unified interface to access the components of :class:`SingletonInputs`.
"""
+ inputs: SingletonInputs
+
+ @cached_property
+ def prompt(self) -> Optional[str]:
+ inputs = self.inputs
+
+ if inputs["type"] == "token" or inputs["type"] == "multimodal":
+ return inputs.get("prompt")
+
+ assert_never(inputs)
+ @cached_property
+ def prompt_token_ids(self) -> List[int]:
+ inputs = self.inputs
+
+ if inputs["type"] == "token" or inputs["type"] == "multimodal":
+ return inputs.get("prompt_token_ids", [])
+
+ assert_never(inputs)
+
+ @cached_property
+ def token_type_ids(self) -> List[int]:
+ inputs = self.inputs
+
+ if inputs["type"] == "token" or inputs["type"] == "multimodal":
+ return inputs.get("token_type_ids", [])
+
+ assert_never(inputs)
+
+ @cached_property
+ def prompt_embeds(self) -> Optional[torch.Tensor]:
+ inputs = self.inputs
+
+ if inputs["type"] == "token" or inputs["type"] == "multimodal":
+ return None
+
+ assert_never(inputs)
+
+ @cached_property
+ def multi_modal_data(self) -> "MultiModalDataDict":
+ inputs = self.inputs
+
+ if inputs["type"] == "token":
+ return inputs.get("multi_modal_data", {})
+
+ if inputs["type"] == "multimodal":
+ return inputs.get("mm_kwargs", {})
+
+ assert_never(inputs)
+
+ @cached_property
+ def multi_modal_inputs(self) -> Union[Dict, "MultiModalKwargs"]:
+ inputs = self.inputs
+
+ if inputs["type"] == "token":
+ return inputs.get("multi_modal_inputs", {})
+
+ if inputs["type"] == "multimodal":
+ return inputs.get("mm_kwargs", {})
+
+ assert_never(inputs)
+
+ @cached_property
+ def multi_modal_placeholders(self) -> "MultiModalPlaceholderDict":
+ inputs = self.inputs
+
+ if inputs["type"] == "token":
+ return inputs.get("multi_modal_placeholders", {})
+
+ if inputs["type"] == "multimodal":
+ return inputs.get("mm_placeholders", {})
+
+ assert_never(inputs)
+
+ @cached_property
+ def mm_processor_kwargs(self) -> Dict[str, Any]:
+ inputs = self.inputs
+
+ if inputs["type"] == "token":
+ return inputs.get("mm_processor_kwargs", {})
+
+ if inputs["type"] == "multimodal":
+ return {}
+
+ assert_never(inputs)
+
+
+ProcessorInputs = Union[DecoderOnlyInputs, EncoderDecoderInputs]
+"""
+The inputs to :data:`vllm.inputs.InputProcessor`.
+"""
_T1 = TypeVar("_T1", bound=SingletonPrompt, default=SingletonPrompt)
_T2 = TypeVar("_T2", bound=SingletonPrompt, default=SingletonPrompt)
@@ -253,10 +377,11 @@ def zip_enc_dec_prompts(
) -> List[ExplicitEncoderDecoderPrompt[_T1, _T2]]:
"""
Zip encoder and decoder prompts together into a list of
- :class:`ExplicitEncoderDecoderPrompt` instances. mm_processor_kwargs
- may also be provided; if a dict is passed, the same dictionary will be
- used for every encoder/decoder prompt. If an iterable is provided, it will
- be zipped with the encoder/decoder prompts.
+ :class:`ExplicitEncoderDecoderPrompt` instances.
+
+ ``mm_processor_kwargs`` may also be provided; if a dict is passed, the same
+ dictionary will be used for every encoder/decoder prompt. If an iterable is
+ provided, it will be zipped with the encoder/decoder prompts.
"""
if mm_processor_kwargs is None:
mm_processor_kwargs = cast(Dict[str, Any], {})
@@ -282,34 +407,3 @@ def to_enc_dec_tuple_list(
return [(enc_dec_prompt["encoder_prompt"],
enc_dec_prompt["decoder_prompt"])
for enc_dec_prompt in enc_dec_prompts]
-
-
-def __getattr__(name: str):
- import warnings
-
- if name == "PromptInput":
- msg = ("PromptInput has been renamed to PromptType. "
- "The original name will be removed in an upcoming version.")
-
- warnings.warn(DeprecationWarning(msg), stacklevel=2)
-
- return PromptType
-
- if name == "LLMInputs":
- msg = ("LLMInputs has been renamed to DecoderOnlyInputs. "
- "The original name will be removed in an upcoming version.")
-
- warnings.warn(DeprecationWarning(msg), stacklevel=2)
-
- return DecoderOnlyInputs
-
- if name == "EncoderDecoderLLMInputs":
- msg = (
- "EncoderDecoderLLMInputs has been renamed to EncoderDecoderInputs. "
- "The original name will be removed in an upcoming version.")
-
- warnings.warn(DeprecationWarning(msg), stacklevel=2)
-
- return EncoderDecoderInputs
-
- raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
diff --git a/flagscale/inference/core/llm_engine.py b/flagscale/inference/core/llm_engine.py
index a765f6cf5..53b6d4a99 100644
--- a/flagscale/inference/core/llm_engine.py
+++ b/flagscale/inference/core/llm_engine.py
@@ -1,4 +1,5 @@
# This file is modified from 'FlagScale/vllm/vllm/engine/llm_engine.py'
+import copy
import time
from collections import Counter as collectionsCounter
from collections import deque
@@ -11,14 +12,12 @@
from typing import Set, Type, Union, cast, overload
import torch
-from typing_extensions import TypeVar
+from typing_extensions import TypeVar, deprecated
import vllm.envs as envs
-from vllm.config import (CacheConfig, DecodingConfig, DeviceConfig,
- EngineConfig, LoadConfig, LoRAConfig, ModelConfig,
- ObservabilityConfig, ParallelConfig,
- PromptAdapterConfig, SchedulerConfig,
- SpeculativeConfig)
+from vllm.config import (DecodingConfig, LoRAConfig, ModelConfig,
+ ObservabilityConfig, ParallelConfig, SchedulerConfig,
+ VllmConfig)
from vllm.core.scheduler import (ScheduledSequenceGroup, Scheduler,
SchedulerOutputs)
from vllm.engine.arg_utils import EngineArgs
@@ -32,8 +31,9 @@
from vllm.executor.executor_base import ExecutorBase
from vllm.executor.gpu_executor import GPUExecutor
from vllm.executor.ray_utils import initialize_ray_cluster
-from vllm.inputs import (INPUT_REGISTRY, DecoderOnlyInputs,
- EncoderDecoderInputs, InputRegistry, PromptType)
+from vllm.inputs import (INPUT_REGISTRY, InputRegistry, ProcessorInputs,
+ PromptType, SingletonInputsAdapter)
+from vllm.inputs.parse import is_encoder_decoder_inputs, is_token_prompt
from vllm.inputs.preprocess import InputPreprocessor
from vllm.logger import init_logger
from vllm.logits_process import get_bad_words_logits_processors
@@ -41,7 +41,8 @@
from vllm.model_executor.guided_decoding import (
get_local_guided_decoding_logits_processor)
from vllm.model_executor.layers.sampler import SamplerOutput
-from vllm.outputs import (EmbeddingRequestOutput, RequestOutput,
+from vllm.multimodal import MULTIMODAL_REGISTRY, MultiModalRegistry
+from vllm.outputs import (PoolingRequestOutput, RequestOutput,
RequestOutputFactory)
from vllm.pooling_params import PoolingParams
from vllm.prompt_adapter.request import PromptAdapterRequest
@@ -81,7 +82,7 @@ def _load_generation_config_dict(model_config: ModelConfig) -> Dict[str, Any]:
_G = TypeVar("_G", bound=BaseTokenizerGroup, default=BaseTokenizerGroup)
-_O = TypeVar("_O", RequestOutput, EmbeddingRequestOutput)
+_O = TypeVar("_O", RequestOutput, PoolingRequestOutput)
@dataclass
@@ -113,7 +114,7 @@ class SchedulerContext:
def __init__(self, multi_step_stream_outputs: bool = False):
self.output_queue: Deque[OutputData] = deque()
self.request_outputs: List[Union[RequestOutput,
- EmbeddingRequestOutput]] = []
+ PoolingRequestOutput]] = []
self.seq_group_metadata_list: Optional[
List[SequenceGroupMetadata]] = None
self.scheduler_outputs: Optional[SchedulerOutputs] = None
@@ -222,30 +223,35 @@ def validate_outputs(
def __init__(
self,
- model_config: ModelConfig,
- cache_config: CacheConfig,
- parallel_config: ParallelConfig,
- scheduler_config: SchedulerConfig,
- device_config: DeviceConfig,
- load_config: LoadConfig,
- lora_config: Optional[LoRAConfig],
- speculative_config: Optional[SpeculativeConfig],
- decoding_config: Optional[DecodingConfig],
- observability_config: Optional[ObservabilityConfig],
- prompt_adapter_config: Optional[PromptAdapterConfig],
+ vllm_config: VllmConfig,
executor_class: Type[ExecutorBase],
log_stats: bool,
usage_context: UsageContext = UsageContext.ENGINE_CONTEXT,
stat_loggers: Optional[Dict[str, StatLoggerBase]] = None,
input_registry: InputRegistry = INPUT_REGISTRY,
+ mm_registry: MultiModalRegistry = MULTIMODAL_REGISTRY,
use_cached_outputs: bool = False,
) -> None:
+
+ self.model_config = vllm_config.model_config
+ self.cache_config = vllm_config.cache_config
+ self.lora_config = vllm_config.lora_config
+ self.parallel_config = vllm_config.parallel_config
+ self.scheduler_config = vllm_config.scheduler_config
+ self.device_config = vllm_config.device_config
+ self.speculative_config = vllm_config.speculative_config # noqa
+ self.load_config = vllm_config.load_config
+ self.decoding_config = vllm_config.decoding_config or DecodingConfig( # noqa
+ )
+ self.prompt_adapter_config = vllm_config.prompt_adapter_config # noqa
+ self.observability_config = vllm_config.observability_config or ObservabilityConfig( # noqa
+ )
+
logger.info(
"Initializing an LLM engine (v%s) with config: "
"model=%r, speculative_config=%r, tokenizer=%r, "
"skip_tokenizer_init=%s, tokenizer_mode=%s, revision=%s, "
- "override_neuron_config=%s, "
- "rope_scaling=%r, rope_theta=%r, tokenizer_revision=%s, "
+ "override_neuron_config=%s, tokenizer_revision=%s, "
"trust_remote_code=%s, dtype=%s, max_seq_len=%d, "
"download_dir=%r, load_format=%s, tensor_parallel_size=%d, "
"pipeline_parallel_size=%d, "
@@ -257,57 +263,46 @@ def __init__(
"num_scheduler_steps=%d, chunked_prefill_enabled=%s "
"multi_step_stream_outputs=%s, enable_prefix_caching=%s, "
"use_async_output_proc=%s, use_cached_outputs=%s, "
- "chat_template_text_format=%s, mm_processor_kwargs=%s)",
+ "mm_processor_kwargs=%s, pooler_config=%r,"
+ "compilation_config=%r",
VLLM_VERSION,
- model_config.model,
- speculative_config,
- model_config.tokenizer,
- model_config.skip_tokenizer_init,
- model_config.tokenizer_mode,
- model_config.revision,
- model_config.override_neuron_config,
- model_config.rope_scaling,
- model_config.rope_theta,
- model_config.tokenizer_revision,
- model_config.trust_remote_code,
- model_config.dtype,
- model_config.max_model_len,
- load_config.download_dir,
- load_config.load_format,
- parallel_config.tensor_parallel_size,
- parallel_config.pipeline_parallel_size,
- parallel_config.disable_custom_all_reduce,
- model_config.quantization,
- model_config.enforce_eager,
- cache_config.cache_dtype,
- model_config.quantization_param_path,
- device_config.device,
- decoding_config,
- observability_config,
- model_config.seed,
- model_config.served_model_name,
- scheduler_config.num_scheduler_steps,
- scheduler_config.chunked_prefill_enabled,
- scheduler_config.multi_step_stream_outputs,
- cache_config.enable_prefix_caching,
- model_config.use_async_output_proc,
+ self.model_config.model,
+ self.speculative_config,
+ self.model_config.tokenizer,
+ self.model_config.skip_tokenizer_init,
+ self.model_config.tokenizer_mode,
+ self.model_config.revision,
+ self.model_config.override_neuron_config,
+ self.model_config.tokenizer_revision,
+ self.model_config.trust_remote_code,
+ self.model_config.dtype,
+ self.model_config.max_model_len,
+ self.load_config.download_dir,
+ self.load_config.load_format,
+ self.parallel_config.tensor_parallel_size,
+ self.parallel_config.pipeline_parallel_size,
+ self.parallel_config.disable_custom_all_reduce,
+ self.model_config.quantization,
+ self.model_config.enforce_eager,
+ self.cache_config.cache_dtype,
+ self.model_config.quantization_param_path,
+ self.device_config.device,
+ self.decoding_config,
+ self.observability_config,
+ self.model_config.seed,
+ self.model_config.served_model_name,
+ self.scheduler_config.num_scheduler_steps,
+ self.scheduler_config.chunked_prefill_enabled,
+ self.scheduler_config.multi_step_stream_outputs,
+ self.cache_config.enable_prefix_caching,
+ self.model_config.use_async_output_proc,
use_cached_outputs,
- model_config.chat_template_text_format,
- model_config.mm_processor_kwargs,
+ self.model_config.mm_processor_kwargs,
+ self.model_config.pooler_config,
+ vllm_config.compilation_config,
)
# TODO(woosuk): Print more configs in debug mode.
- self.model_config = model_config
- self.cache_config = cache_config
- self.lora_config = lora_config
- self.parallel_config = parallel_config
- self.scheduler_config = scheduler_config
- self.device_config = device_config
- self.speculative_config = speculative_config
- self.load_config = load_config
- self.decoding_config = decoding_config or DecodingConfig()
- self.prompt_adapter_config = prompt_adapter_config
- self.observability_config = observability_config or ObservabilityConfig(
- )
+
self.log_stats = log_stats
self.use_cached_outputs = use_cached_outputs
@@ -329,27 +324,17 @@ def get_tokenizer_for_seq(sequence: Sequence) -> AnyTokenizer:
self.seq_counter = Counter()
self.generation_config_fields = _load_generation_config_dict(
- model_config)
+ self.model_config)
- self.input_preprocessor = InputPreprocessor(model_config,
- self.tokenizer)
+ self.input_preprocessor = InputPreprocessor(self.model_config,
+ self.tokenizer,
+ mm_registry)
self.input_registry = input_registry
self.input_processor = input_registry.create_input_processor(
- model_config)
-
- self.model_executor = executor_class(
- model_config=model_config,
- cache_config=cache_config,
- parallel_config=parallel_config,
- scheduler_config=scheduler_config,
- device_config=device_config,
- lora_config=lora_config,
- speculative_config=speculative_config,
- load_config=load_config,
- prompt_adapter_config=prompt_adapter_config,
- observability_config=self.observability_config,
- )
+ self.model_config)
+
+ self.model_executor = executor_class(vllm_config=vllm_config, )
if self.model_config.task != "embedding":
self._initialize_kv_caches()
@@ -359,36 +344,36 @@ def get_tokenizer_for_seq(sequence: Sequence) -> AnyTokenizer:
from vllm.model_executor.model_loader import (
get_architecture_class_name)
usage_message.report_usage(
- get_architecture_class_name(model_config),
+ get_architecture_class_name(self.model_config),
usage_context,
extra_kvs={
# Common configuration
"dtype":
- str(model_config.dtype),
+ str(self.model_config.dtype),
"tensor_parallel_size":
- parallel_config.tensor_parallel_size,
+ self.parallel_config.tensor_parallel_size,
"block_size":
- cache_config.block_size,
+ self.cache_config.block_size,
"gpu_memory_utilization":
- cache_config.gpu_memory_utilization,
+ self.cache_config.gpu_memory_utilization,
# Quantization
"quantization":
- model_config.quantization,
+ self.model_config.quantization,
"kv_cache_dtype":
- str(cache_config.cache_dtype),
+ str(self.cache_config.cache_dtype),
# Feature flags
"enable_lora":
- bool(lora_config),
+ bool(self.lora_config),
"enable_prompt_adapter":
- bool(prompt_adapter_config),
+ bool(self.prompt_adapter_config),
"enable_prefix_caching":
- cache_config.enable_prefix_caching,
+ self.cache_config.enable_prefix_caching,
"enforce_eager":
- model_config.enforce_eager,
+ self.model_config.enforce_eager,
"disable_custom_all_reduce":
- parallel_config.disable_custom_all_reduce,
+ self.parallel_config.disable_custom_all_reduce,
})
if self.tokenizer:
@@ -407,7 +392,7 @@ def get_tokenizer_for_seq(sequence: Sequence) -> AnyTokenizer:
for _ in range(self.parallel_config.pipeline_parallel_size)
]
- if model_config.use_async_output_proc:
+ if self.model_config.use_async_output_proc:
process_model_outputs = weak_bind(self._process_model_outputs)
self.async_callbacks = [
@@ -427,11 +412,11 @@ def get_tokenizer_for_seq(sequence: Sequence) -> AnyTokenizer:
# GPU and CPU blocks, which are profiled in the distributed executor.
self.scheduler = [
Scheduler(
- scheduler_config, cache_config, lora_config,
- parallel_config.pipeline_parallel_size,
+ self.scheduler_config, self.cache_config, self.lora_config,
+ self.parallel_config.pipeline_parallel_size,
self.async_callbacks[v_id]
- if model_config.use_async_output_proc else None)
- for v_id in range(parallel_config.pipeline_parallel_size)
+ if self.model_config.use_async_output_proc else None)
+ for v_id in range(self.parallel_config.pipeline_parallel_size)
]
# Metric Logging.
@@ -453,7 +438,8 @@ def get_tokenizer_for_seq(sequence: Sequence) -> AnyTokenizer:
"prometheus":
PrometheusStatLogger(
local_interval=_LOCAL_LOGGING_INTERVAL_SEC,
- labels=dict(model_name=model_config.served_model_name),
+ labels=dict(
+ model_name=self.model_config.served_model_name),
max_model_len=self.model_config.max_model_len),
}
self.stat_loggers["prometheus"].info("cache_config",
@@ -506,7 +492,7 @@ def _initialize_kv_caches(self) -> None:
@classmethod
def _get_executor_cls(cls,
- engine_config: EngineConfig) -> Type[ExecutorBase]:
+ engine_config: VllmConfig) -> Type[ExecutorBase]:
distributed_executor_backend = (
engine_config.parallel_config.distributed_executor_backend)
# Initialize the cluster and specify the executor class.
@@ -533,6 +519,14 @@ def _get_executor_cls(cls,
elif engine_config.device_config.device_type == "cpu":
from vllm.executor.cpu_executor import CPUExecutor
executor_class = CPUExecutor
+ elif engine_config.device_config.device_type == "hpu":
+ if distributed_executor_backend == "ray":
+ initialize_ray_cluster(engine_config.parallel_config)
+ from vllm.executor.ray_hpu_executor import RayHPUExecutor
+ executor_class = RayHPUExecutor
+ else:
+ from vllm.executor.hpu_executor import HPUExecutor
+ executor_class = HPUExecutor
elif engine_config.device_config.device_type == "openvino":
from vllm.executor.openvino_executor import OpenVINOExecutor
executor_class = OpenVINOExecutor
@@ -576,11 +570,11 @@ def from_engine_args(
) -> "LLMEngine":
"""Creates an LLM engine from the engine arguments."""
# Create the engine configs.
- engine_config = engine_args.create_engine_config()
+ engine_config = engine_args.create_engine_config(usage_context)
executor_class = cls._get_executor_cls(engine_config)
# Create the LLM engine.
engine = cls(
- **engine_config.to_dict(),
+ vllm_config=engine_config,
executor_class=executor_class,
log_stats=not engine_args.disable_log_stats,
usage_context=usage_context,
@@ -627,7 +621,7 @@ def _init_tokenizer(self) -> BaseTokenizerGroup:
model_config=self.model_config,
scheduler_config=self.scheduler_config,
parallel_config=self.parallel_config,
- enable_lora=bool(self.lora_config))
+ lora_config=self.lora_config)
def _verify_args(self) -> None:
self.model_config.verify_with_parallel_config(self.parallel_config)
@@ -643,7 +637,7 @@ def _verify_args(self) -> None:
def _add_processed_request(
self,
request_id: str,
- processed_inputs: Union[DecoderOnlyInputs, EncoderDecoderInputs],
+ processed_inputs: ProcessorInputs,
params: Union[SamplingParams, PoolingParams],
arrival_time: float,
lora_request: Optional[LoRARequest],
@@ -668,38 +662,45 @@ def _add_processed_request(
)
return None
- self._validate_model_inputs(processed_inputs)
+ self._validate_model_inputs(processed_inputs, lora_request)
# Create the sequences.
block_size = self.cache_config.block_size
seq_id = next(self.seq_counter)
eos_token_id = self.input_preprocessor.get_eos_token_id(lora_request)
- seq = Sequence(seq_id, processed_inputs, block_size, eos_token_id,
- lora_request, prompt_adapter_request)
+ negative_decoder_inputs = None # --- FLAGSCALE MODIFICATION ---
+ if is_encoder_decoder_inputs(processed_inputs):
+ decoder_inputs = processed_inputs["decoder"]
+ encoder_inputs = processed_inputs["encoder"]
+ # --- FLAGSCALE MODIFICATION BEG ---
+ elif "negative_prompt_token_ids" in processed_inputs \
+ and processed_inputs["negative_prompt_token_ids"] is not None:
+ positive_inputs = processed_inputs.copy()
+ positive_inputs.pop("negative_prompt_token_ids")
+ negative_inputs = processed_inputs.copy()
+ negative_inputs["prompt_token_ids"] = negative_inputs["negative_prompt_token_ids"]
+ negative_inputs.pop("negative_prompt_token_ids")
+ decoder_inputs = positive_inputs
+ negative_decoder_inputs = negative_inputs
+ encoder_inputs = None
+ # --- FLAGSCALE MODIFICATION END ---
+ else:
+ decoder_inputs = processed_inputs
+ encoder_inputs = None
- encoder_seq = None
- if 'encoder_prompt_token_ids' in processed_inputs:
- encoder_seq = Sequence(seq_id,
- processed_inputs,
- block_size,
- eos_token_id,
- lora_request,
- prompt_adapter_request,
- from_decoder_prompt=False)
+ seq = Sequence(seq_id, decoder_inputs, block_size, eos_token_id,
+ lora_request, prompt_adapter_request)
# --- FLAGSCALE MODIFICATION BEG ---
- negative_seq = None
- if "negative_prompt_token_ids" in processed_inputs \
- and processed_inputs["negative_prompt_token_ids"]:
- negative_seq = Sequence(seq_id,
- processed_inputs,
- block_size,
- eos_token_id,
- lora_request,
- prompt_adapter_request,
- from_negative_prompt=True)
+ negative_seq = (None if negative_decoder_inputs is None else Sequence(
+ seq_id, negative_decoder_inputs, block_size, eos_token_id, lora_request,
+ prompt_adapter_request))
# --- FLAGSCALE MODIFICATION END ---
+ encoder_seq = (None if encoder_inputs is None else Sequence(
+ seq_id, encoder_inputs, block_size, eos_token_id, lora_request,
+ prompt_adapter_request))
+
# Create a SequenceGroup based on SamplingParams or PoolingParams
if isinstance(params, SamplingParams):
seq_group = self._create_sequence_group_with_sampling(
@@ -740,7 +741,8 @@ def _add_processed_request(
def stop_remote_worker_execution_loop(self) -> None:
self.model_executor.stop_remote_worker_execution_loop()
- @overload # DEPRECATED
+ @overload
+ @deprecated("'inputs' will be renamed to 'prompt")
def add_request(
self,
request_id: str,
@@ -841,9 +843,21 @@ def add_request(
raise ValueError(f"Got priority {priority} but "
"Priority scheduling is not enabled.")
+ if isinstance(params, SamplingParams) \
+ and (params.guided_decoding or params.logits_processors) \
+ and self.scheduler_config.num_scheduler_steps > 1:
+ raise ValueError(
+ "Guided decoding and logits processors are not supported "
+ "in multi-step decoding")
+
if arrival_time is None:
arrival_time = time.time()
+ if self.tokenizer is not None:
+ self._validate_token_prompt(
+ prompt,
+ tokenizer=self.get_tokenizer(lora_request=lora_request))
+
preprocessed_inputs = self.input_preprocessor.preprocess(
prompt,
request_id=request_id,
@@ -852,13 +866,6 @@ def add_request(
)
processed_inputs = self.input_processor(preprocessed_inputs)
- # This is a bit of a hack - copy the mm_processor_kwargs that were
- # used in the input processor to the processed output, since these
- # kwargs are presumed to be immutable and the values should be aligned
- # between the input processor (here) and the input mapper.
- processed_inputs["mm_processor_kwargs"] = preprocessed_inputs.get(
- "mm_processor_kwargs")
-
self._add_processed_request(
request_id=request_id,
processed_inputs=processed_inputs,
@@ -870,6 +877,27 @@ def add_request(
priority=priority,
)
+ def _validate_token_prompt(self, prompt: PromptType,
+ tokenizer: AnyTokenizer):
+ # Guard against out-of-vocab tokens.
+ # For some tokenizers, tokenizer.decode will happily return empty text
+ # for token ids that are out of vocab, and we don't detect token ids
+ # that are greater than the max token id before running the model.
+ # However, these token ids will later crash a cuda kernel at runtime
+ # with an index out of bounds error. This will crash the entire engine.
+ # This needs to happen before multimodal input pre-processing, which
+ # may add dummy Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.
- -Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:
-[Bugfix] for bug fixes.[CI/Build] for build or continuous integration improvements.[Doc] for documentation fixes and improvements.[Model] for adding a new model or improving an existing model. Model name should appear in the title.[Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.) [Kernel] for changes affecting CUDA kernels or other compute kernels.[Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)[Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).[Misc] for PRs that do not fit the above categories. Please use this sparingly.Note: If the PR spans more than one category, please include all relevant prefixes.
- -The PR need to meet the following code quality standards:
- -format.sh to format your code.docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.Each custom kernel needs a schema and one or more implementations to be registered with PyTorch.
-Tensors require meta-functions. Meta-functions should be implemented and registered in python so that dynamic dims can be handled automatically. See above documents for a description of meta-functions.torch.libary.opcheck() to test the function registration and meta-function for any registered ops. See tests/kernels for examples.Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.
The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:
- - action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!
- - -