Skip to content

Commit

Permalink
Update interval_encoding.rs
Browse files Browse the repository at this point in the history
  • Loading branch information
aneubeck committed Jul 16, 2024
1 parent 94f5cc3 commit 9ff1ffa
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion crates/bpe/src/interval_encoding.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ use crate::byte_pair_encoding::BytePairEncoding;
/// is the predecessor of another node.
/// With the `tree_depth` field the number of path length (which is equivalent to the number of encoded tokens) can be determined
/// in O(1) as well.
///
///
/// Note: the fields `tree_end` and `tree_depth` could also be represented by succinct data structures, reducing their size drastically.
/// Since we still need the `tree_id` and `last_token` fields, this would in total reduce memory footprint by a bit less than 50%.
pub struct IntervalEncoding<'a> {
Expand Down

0 comments on commit 9ff1ffa

Please sign in to comment.