llama: add support for QRWKV6 model architecture (#11001)
llama: add support for QRWKV6 model architecture (#11001)
* WIP: Add support for RWKV6Qwen2
Signed-off-by: Molly Sophia <[email protected]>
* RWKV: Some graph simplification
Signed-off-by: Molly Sophia <[email protected]>
* Add support for RWKV6Qwen2 with cpu and cuda GLA
Signed-off-by: Molly Sophia <[email protected]>
* RWKV6[QWEN2]: Concat lerp weights together to reduce cpu overhead
Signed-off-by: Molly Sophia <[email protected]>
* Fix some typos
Signed-off-by: Molly Sophia <[email protected]>
* code format changes
Signed-off-by: Molly Sophia <[email protected]>
* Fix wkv test & add gla test
Signed-off-by: Molly Sophia <[email protected]>
* Fix cuda warning
Signed-off-by: Molly Sophia <[email protected]>
* Update README.md
Signed-off-by: Molly Sophia <[email protected]>
* Update ggml/src/ggml-cuda/gla.cu
Co-authored-by: Georgi Gerganov <[email protected]>
* Fix fused lerp weights loading with RWKV6
Signed-off-by: Molly Sophia <[email protected]>
* better sanity check skipping for QRWKV6 in llama-quant
thanks @compilade
Signed-off-by: Molly Sophia <[email protected]>
Co-authored-by: compilade <[email protected]>
---------
Signed-off-by: Molly Sophia <[email protected]>
Co-authored-by: Georgi Gerganov <[email protected]>
Co-authored-by: compilade <[email protected]>
