Add Bamba Model

[fabianlim]

This is the companion PR to an huggingface PR for adding Bamba, which is a hybrid mamba2 architecture with SwiGLU. The checkpoints are jointly trained by IBM, Princeton, and UIUC.

In this PR we have:

• Created the bamba model inference architecture, which we would like acknowledge the jamba team for referencing their implementation, whereby we modified to support full attention layers with RoPE and mamba v2.
• Ensured that we have TP support.
• Ensured we support chunked prefill. Currently we have a partial solution, which works only when the cont batch boundaries line up with the chunked boundaries. This is now completely fixed.
• Ensured that we conform to the recent PR for adding pipeline support for SSM models.
• Adapted the mamba v2 scan kernels into vllm/model_executor/layers/mamba/ops. Only the fwd kernels are extracted. Some modifications and fixes are made.
• created tests/models/decoder_only/language/test_bamba.py with an initial ibm-fms/Bamba-9.8b-1.8T-hf. This is practically identical to test_mamba.py, only chunked prefill tests are disabled as it is currently not supported.

Currently only FlashAttention backend is supported, as we check fields like context_lens_tensor. Have not yet investigated other backends.

We would like to also acknowledge the draft codestral mamba PR from @tlrmchlsmth, which we also referenced the mixer.

• we made a few simplications for bamba (simplified mixer from mamba v2)
• Cuda graph capturing seems to be working, but we understand that cudagraphs are disabled for long sequence lengths. For SSM models the strength is in this regime, so can we handle it better?

Hope to discuss the following with the maintainers

1. do we have to remove all the bwd kernels? yes we should
2. for the full attention layers, we increase the sin_cos cache to cover the sequence length, if it is longer than max_sequence_len. This differs for other current models (e.g., llama). How can we better support long sequence lengths? we should keep this consistent with other models, so we propose to allow the sin_cos cache extension only when VLLM_ALLOW_LONG_MAX_MODEL_LEN is specified.
3. have some ideas to support chunked pre-fill, but will appreciate some discussion with the maintainers on how to proceed. working on changing the kernels to support chunked prefill.
4. since the mixer2 is simplified from mamba, should we rename it? we can keep it as is, but we should document the differences from mamba_ssm

cc: @ani300, @raghukiran1224, @cyang49, @njhill

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[tlrmchlsmth]

Hi @fabianlim, thanks for the PR! It's really great to see progress being made on state-space models, especially for me as I unfortunately haven't been able to prioritize support for Mamba2

I'm happy to shepherd this PR and discuss any questions you have, especially to support chunked prefill. If you haven't already, can you join the developer slack for quicker discussion? (https://communityinviter.com/apps/vllm-dev/join-vllm-developers-slack)

[fabianlim]

@tlrmchlsmth I cleaned up the PR quite abit, perhaps it might be a good time to get some early eyes. The chunked prefill implementation is incomplete ATM, as we discussed offline.

[tlrmchlsmth]

first pass, just a few comments. At a high level it looks good.

Will you add a test for tensor parallelism?

[tlrmchlsmth]

The comment trails off, but will there be a small test model available?

[fabianlim]

@raghukiran1224 any plans for a small test model? I think since we do outputs comparison it is not that good to just have a randomly initialised small model

[raghukiran1224]

@fabianlim @tlrmchlsmth would it be ok to test with a random model or would you rather have a tiny model (say 200M or so) to test with?

[tlrmchlsmth]

A tiny model with nonrandom weights would be much better!

[tlrmchlsmth]

btw is there any update on this?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fabianlim.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[fabianlim]

@tlrmchlsmth i have addressed most of your comments now, not rebasing yet, waiting for you to look first. But I realized test_jamba.py has changed so I will need to do the rename and test again.

[tlrmchlsmth]

@fabianlim At a high level, the changes look good, and the PR looks good overall. I'll do a more thorough review once it's unmarked as draft.

Could you add unit tests for the added kernels in layers/mamba/ops?

[fabianlim]

@tlrmchlsmth I have now marked the PR as ready and have addressed the remaining items, in particular the chunked prefill is the biggest change. This requires changes into the kernels. To test this, I have added unit tests for various chunked prefill scenarios.

However in test_mamba_prefill_chunking I removed two prompts due to difficulties getting the decoding results to exactly match; the unit tests are quite extensive though and the match was quite statisfactory there. The original tests for jamba also removed some prompts for the same reason.

In the unit tests, I have things like [(8, 8), (8, 8), (8, 8)], this tests the chunk prefill as follows

• the first (cont batch) is 8 tokens of example 1, followed by 8 tokens of example 2, after the decoding the state is saved.
• the second (cont batch) is 8 tokens of example 1, followed by 8 tokens of example 2. To ensure correct decoding, the saved state must be passed into the kernels (the updates to the kernels are to ingest this state data appropriately).
• and so on ...

There are other tests that vary the chunk size, the number of tokens being passed in each batch, etc..

Know is the holiday season and pls take your time. Happy holidays!

[tlrmchlsmth]

@fabianlim do you know when a version of transformers that supports bamba will be released? This PR will likely be blocked until it's out

[tlrmchlsmth]

btw is there any update on this?

[tlrmchlsmth]

Instead of this, have you considered using rope_scaling instead? If you use get_rope instead of constructing the RotaryEmbedding directly, I think it should work for bamba

See this unit test for an example of how it works:
https://github.com/sasha0552/vllm/blob/d427e5cfda8d2536b81e6021128e71b2dbc281aa/tests/test_config.py#L177

See also:
https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/

[fabianlim]

@tlrmchlsmth ok I will follow your suggestion. this has been changed in 63f5340

[fabianlim]

hey @tlrmchlsmth thanks for your comments will get to them, the model has been merged to HF main, but it is still pending an official release to my knowledge. Hopefully that should happen soon since the last patch release was 2 weeks ago.

I addressed most of the comments in the following commits. I have replaced with get_rope, however I am a little confused with the dynamic implementation. I cant seem to understand how the cache is dynamic; it seems like DynamicNTKScalingRotaryEmbedding indeed has a different alpha, but the cache is statically computed for the "largest it can ever be" and on init. However the point is that it is still static, as "largest" is still determined on init. However, we did this to get a dynamic cache that adjusts whenever longer sequences come in (which may not be known on init).

[fabianlim]

@tlrmchlsmth ok I have updated the PR again where I have addressed the last remaining comment on the rope behavior, the team is in the process of finalizing the dev checkpoint, when we upload it will ping you again.

[tlrmchlsmth]

@fabianlim Thanks, sounds good!