Adding cascade inference to vLLM

[raywanb]

This pr adds cascade inference to vllm. Related to #8132 It should speedup inference when there are multiple requests that share the same prefix with a cold cache. Currently, the second level of cascade inference is not turned on yet. This is task 1 of milestone 1 for #8786.

Benchmarking Results Can be Found Here:
https://docs.google.com/spreadsheets/d/13d9m0iHKuWotKv_euTYNmEM6eyRDlvlyUqTuTosxZkY/edit?usp=sharing

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[comaniac]

@elfiegg could you also help review this PR, and provide some suggestions to @raywanb about how to add chunked prefill with the cascade kernel?

[elfiegg]

Adding @yzh119 for comments too.

[elfiegg]

Hey @raywanb thanks for the updates! With these new changes, it looks like the cascade inference will replace the old batch prefill/decode kernel API calls. Before we dive into the implementation details, I'd love to overcommunicate on the following:

• Backward compatibility:
  • How will compatibility be maintained, especially for FlashInfer 1.5 and earlier versions (for which I assume don't support cascade inference)?
  • What’s the plan for features like multi-step, and speculative decoding, sliding window, etc?
• Performance gains:
  • If the main motivation for this integration is performance, do we have performance data measured on key models without beam search?

[raywanb]

Hey @elfiegg,

Sorry, for the late reply.

1. We will probably do some version gating to ensure FlashInfer compatibility .
2. This PR won't add support for them. Future pr may support them.
3. Lastly, we do have some benchmark data on kernel time for beam search workloads compared to existing kernel, but will test for performance on key models later.

Thanks!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @raywanb.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[pavanimajety]

Do all models have the same naming convention for the scale attribute? Is it better to have a default value like 1.0 and a remark/warning instead of throwing an error?

[raywanb]

not sure if all models have that attribute. Updated the code to throw warning and a default value of None.

[pavanimajety]

Same as above

[raywanb]

as above