Xmixers: A collection of SOTA efficient token/channel mixers

💬 Discord •

Introduction

This repository aims to implement SOTA efficient token/channel mixers. Any technologies related to non-Vanilla Transformer are welcome. If you are interested in this repository, please join our Discord.

Roadmap

• Token Mixers
  • Linear Attention
  • Linear RNN
  • Long Convolution
• Channel Mixers

Pretrained weights

• GPT
  • Doreamonzzz/xmixers_gpt_120m_50b
• LLaMA
  • Doreamonzzz/xmixers_llama_120m_50b

ToDo

• Add special init.

Model

• LLaMA.
• GPT.

Basic

• Add data type for class and function.

Ops

• long_conv_1d_op.

Token Mixers

• Gtu.

Note

[Feature Add]
[Bug Fix]
[Benchmark Add]
[Document Add]
[README Add]