Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Implement LayerSkip #2

Draft
wants to merge 19 commits into
base: paper_default
Choose a base branch
from
Draft

[WIP] Implement LayerSkip #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
68e83ea
set probabilities to be a list
mostafaelhoushi Jul 8, 2024
671779d
use setter wrapper for drop_p
mostafaelhoushi Jul 8, 2024
f3f3591
implement layer wise schedule for dropout
mostafaelhoushi Jul 9, 2024
1c77e33
ensure layer drop condition works
mostafaelhoushi Jul 11, 2024
1900596
start early_exit_loss.py
mostafaelhoushi Jul 11, 2024
3d806c4
get early exit loss to work with seamless
mostafaelhoushi Jul 16, 2024
e138eae
fix exit hooks
mostafaelhoushi Jul 16, 2024
3e8cd67
add profiling code for SamplingGenerator
mostafaelhoushi Aug 27, 2024
baebda3
add other loss scaling types and make uniform default
mostafaelhoushi Aug 27, 2024
dcaf66e
start speculative decoding
mostafaelhoushi Aug 29, 2024
9c4f52b
end-to-end speculative decoding but not correct
mostafaelhoushi Aug 31, 2024
3721aca
create functions for prefill for draft model
mostafaelhoushi Aug 31, 2024
5a2e021
cleanup and get draft model to actually decode
mostafaelhoushi Aug 31, 2024
8aa1a91
verify k_speculate tokens
mostafaelhoushi Aug 31, 2024
5480af5
remove scores calculation for draft prefill
mostafaelhoushi Sep 1, 2024
8331df2
make verification step return vocab
mostafaelhoushi Sep 1, 2024
a844e99
calculate q p as per gpt-fast speculative decoding algorithm
mostafaelhoushi Sep 1, 2024
4ac712b
fix shapes of tensors
mostafaelhoushi Sep 1, 2024
e6ce2e7
add more changes
mostafaelhoushi Sep 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/fairseq2/generation/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,11 @@
from fairseq2.generation.sampling import Sampler as Sampler
from fairseq2.generation.sampling import (
SamplingSeq2SeqGenerator as SamplingSeq2SeqGenerator,
SpeculativeSamplingSeq2SeqGenerator as SpeculativeSamplingSeq2SeqGenerator,
)
from fairseq2.generation.sampling import (
SamplingSequenceGenerator as SamplingSequenceGenerator,
SpeculativeSamplingSequenceGenerator as SpeculativeSamplingSequenceGenerator,
)
from fairseq2.generation.sampling import TopKSampler as TopKSampler
from fairseq2.generation.sampling import TopPSampler as TopPSampler
Expand Down
Loading