Roadmap

[tjtanaa]

1. Port vllm/main feature to ROCm

• Support Llama/Llama-2 models for v0.2.x
• Support SqueezeLLM
• Support YARN
• Merge into upstream vllm ([Continuation] Merge EmbeddedLLM/vllm-rocm into vLLM main vllm-project/vllm#1836)
• Look into supporting multi LORA on ROCm (Add multi-LoRA support vllm-project/vllm#1804)
• Support GGML Kernel (GGUF Quantization on ROCm) (https://github.com/EmbeddedLLM/vllm/tree/ggml-rocm) ([Kernel][Hardware][AMD] Add support for GGUF quantization on ROCm vllm-project/vllm#10254)
• Prompt https://github.com/LMCache/LMCache
  • Add ROCm support to torchac_cuda (https://github.com/EmbeddedLLM/torchac_rocm)
  • Validate rocTX usage in Python (skip for now)
• Support AQLM Kernel (https://github.com/EmbeddedLLM/vllm/tree/aqlm-rocm)
• Upstream Cross-Attention kernel to support Llama 3.2 Vision Model
  • BLOCKER: When passing only text input, the LLM engine will crash.

2. Upstream New Feature

• Add context parallelism support through Star-Attention

4. Benchmark

• Real-world Distribution benchmark https://blog.vllm.ai/2024/10/23/vllm-serving-amd.html
• Benchmark GGUF support
• Long-Input-Long-Output benchmarking.

Interesting works:

• https://github.com/xiayuqing0622/customized-flash-attention

Interesting features regarding:

1. long context mechanism on vLLM:

• [Bugfix]Fix evict v2 with long context length vllm-project/vllm#5411
• [Model] Implement DualChunkAttention for Qwen2 Models vllm-project/vllm#6139

2. compute kernels on vLLM:

• flashinfer paged attention vllm-project/vllm#2772

3. quantization schemes on vLLM:

• [Kernel][Quantization] Custom Floating-Point Runtime Quantization vllm-project/vllm#8751

4. Disaggregated prefill feature on vLLM:

• https://github.com/vllm-project/vllm/pull/10884/files
• https://github.com/kvcache-ai/Mooncake

5. vLLM v0.7.0 tracker [Release]: v0.7.0 Release Tracker vllm-project/vllm#11218

[HAN-oQo]

Hi, @tjtanaa
I wonder how the roadmap is going on.
I quite excited to use AWQ quantized format, when can it be supported?

[tjtanaa]

Hi, @tjtanaa I wonder how the roadmap is going on. I quite excited to use AWQ quantized format, when can it be supported?

@HAN-oQo Hi, vLLM authors said they are working on more efficient AWQ implementation on triton. So, we will address the AWQ on ROCm after they have released their new kernel.

[HAN-oQo]

Thank you for answer! @tjtanaa
I also wonder why safetensor format is not supported, and do you have a plan to support it!

Thank you for offering the nice project.

[tjtanaa]

Thank you for answer! @tjtanaa I also wonder why safetensor format is not supported, and do you have a plan to support it!

Thank you for offering the nice project.

@HAN-oQo The loading of safetensors is buggy on ROCm platform. The memory management during loading of safetensors might be causing the issue on ROCm platform. It often encounters this issue when tensor-parallelism is larger than 1; however, loading from pt is totally fine.