[Core] Support global prefix caching

[lyppg]

An extension of APC to implement global prefix cache

Global prefix cache can be useful in the following use cases:

1. When the APC cache is evicted and can't hit, the processing of new prompts in prefill phase can reuse the KV cache in global KV cache pool and skip the re-computation
2. Another vllm instance can use the KV cache in global KV cache pool directly even if it runs the prompts the first time(which means no APC cache available yet)

This PR extends APC to implement global prefix cache. It simply uses a local Dict as the global prefix KV cache pool, write to it when KV cache computing in prefill phase is done and read from it when updating input_tokens in model_runner. Currently the implementation is simple, it doesn't involve model/cuda change and I can observe some performance improvement in my environment. I tested with some papers(10-40KB) on dataset Long-document-dataset on L4, compared with vanilla vllm, APC+this PR reduces the generation time around 10%~28% with the same result.

Next Steps

In theory it should work better when the prompt is longer, based on the assumption that CPU->GPU copy is faster than GPU KV cache re-computation in prefill phase, but I will do more testing on other datasets and hardware. The cpu<->gpu memory copy can be optimized to improve performance. It can also be integrated with other distributed KV cache pool projects. Please leave comments and feedbacks, thanks.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀