Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Omni-Attention Implementation #32

Open
ChocoWu opened this issue Sep 18, 2024 · 1 comment
Open

Omni-Attention Implementation #32

ChocoWu opened this issue Sep 18, 2024 · 1 comment

Comments

@ChocoWu
Copy link

ChocoWu commented Sep 18, 2024

Great Great work.
I'm very interested in Omni-Attention, but I couldn't find it in the code. Could you please provide a hint or some explanation about Omni-Attention?
@Sierkinhane
Copy link
Collaborator

Hi, we use different mask generation functions here. Here is the latest implementation of Omni-Attention based on FlexAttention. Stay tuned and I will complete it asap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ChocoWu @Sierkinhane