Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

anchor reweighting on GPT-J #10

Open
Tincsvsv opened this issue Dec 26, 2023 · 1 comment
Open

anchor reweighting on GPT-J #10

Tincsvsv opened this issue Dec 26, 2023 · 1 comment

Comments

@Tincsvsv
Copy link

Hello, if I apply anchor reweighting on GPT-J, what parts should I modify? What are the hardware requirements? Looking forward to your reply, thank you very much!
@leanwang326
Copy link
Collaborator

adapting some code in attentioner_for_train.py, i.e. adding gptj_attn and GPTJattentionermanager (like those in attentioner_for_attribution.py) should be sufficient.
For hardware requirements, you might need to set GPT-J to float16 to run in a 40GB gpu. (I have not try this before, that's just an estimation)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@leanwang326 @Tincsvsv