More presto benchmarking #384
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
ivanzvonkov
commented
Feb 6, 2024
| @@ -455,7 +457,7 @@ def forward( | |||
| if aggregate == Aggregate.MEAN: | |||
| return self.norm(x.mean(dim=1)) | |||
| elif aggregate == Aggregate.BAND_GROUPS_MEAN: | |||
| return self.norm(self.band_groups_mean(x, kept_indices, num_timesteps)) | |||
There was a problem hiding this comment.
Not possible to use self.norm here as it expected embedding_size shape
ivanzvonkov
commented
Feb 6, 2024
| @@ -358,7 +358,9 @@ def band_groups_mean( | |||
| mask = (kept_indices >= min_idx) & (kept_indices < max_idx) | |||
| # we assume kept_elements is the same for all batches | |||
| kept_elements = sum(mask[0, :]) | |||
| groups.append(x[mask.bool()].view(batch_size, kept_elements, embedding_dim).mean(dim=1)) | |||
| one_group = x[mask.bool()].view(batch_size, kept_elements, embedding_dim).mean(dim=1) | |||
| one_group_normed = self.norm(one_group) | |||
There was a problem hiding this comment.
My fix for the self.norm issue. Creating a new LayerNorm was not possible because then it expected a weight and bias for that variable in the pretrained model.
gabrieltseng
reviewed
Feb 7, 2024
| @@ -9,7 +9,7 @@ | |||
| "\n", | |||
There was a problem hiding this comment.
Lots of convergence warnings here; I wonder if its worth increasing the number of iterations just to see.
Reply via ReviewNB
There was a problem hiding this comment.
Yes! This is why I used the StandardScaler as attempt at a solution. I'll try more iterations
gabrieltseng
approved these changes
Feb 7, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ran benchmark using Presto band group encodings.
Posting Google sheet in slack.