Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding packed bert from optimum-main #71

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

arsalanu
Copy link

@arsalanu arsalanu commented Mar 9, 2023

Adding PackedBERT notebooks/ models/ utils folder into Paperspace from HF Optimum:

------ copied from the original PR description for Hugging Face Optimum Graphcore (now merged):

Contents:
Simplified notebooks for all three supported Packed BERT tasks for easy implementation
Adds all of the necessary utils/model heads imported into notebooks - preprocessing, postprocessing, model changes

Notes: For the time being, the models/ and utils/ are in this folder that goes into notebooks/ but ideally it would be nice to have the utils put into optimum/graphcore/ so they could be easily importable with the package - and the models/modeling_bert_packed.py could just be options within the default modeling_bert, and packing could be enabled through the AutoConfig (some tweaks would be needed for that, but nothing extensive) This also gives us a structure to add future packing tasks/notebooks

Fixes
I've removed the model classes and packing algorithm/dataset creation utils from the notebooks, noted that they were too complex and large as notebook code blocks requiring too much explanation and would be hard to maintain here. The intention of these notebooks is to give brief explanations of the differences between unpacked and packed at each stage and allows users to easily implement it using the importable methods.
A more in depth explanation of the packing/preproc/postproc/model change process will get its own notebooks/blog in future so we don't need to cover it for this notebook
I've used the env variables for pod type and executable dir
Rewritten most of these notebooks to not be as detailed/complex and use more active language - some of it is copied from existing notebooks for the same (unpacked) tasks - happy to change stuff

@anjleeg-gcai anjleeg-gcai marked this pull request as draft March 10, 2023 11:48
@arsalanu
Copy link
Author

I think this is ready

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants