Initial release

[hendrikvanantwerpen]

The bpe crate names was released, so I released an initial version of the crate to claim the name.

Things changed:

• Added some fields to the crate manifest.
• Changed the serialization of token dictionaries. The problem was that <crates.io> has a size limit of 10MB, while our serialized BPE instances were around 15MB and 30MB. For now I've opted to serialize the token lists + hash factor, and build the BPE instance in the lazy function. The performance impact of this is only relevant when the values are initialized, which is only once per run. But I'm happy to iterate ont his if necessary.

I decided to go ahead with the release to make sure we got the name. I imagine we do another release soon with additional changes or polishing we think are necessary.

[aneubeck]

The reason why I added the serialized representation in the first place was because building the aho corasick data structures from scratch takes several seconds!

I.e. this is a cost, we don't want to pay when running blackbird tests (or when other ppl depend on this).
One option could be to improve the aho-corasick serialization, since we have our own fork anyways. But, I didn't check how much data it actually has. It might just be tricky to get it small, since with have 100k resp. 200k tokens in it. And you need to store probably a couple of numbers for every character in them. So, making it small might be challenging...

[aneubeck]

I looked at the serialization code of the daaghorse crate and it doesn't look like we can safe a ton there.
The only thing which might work is to run gzip over the serialized representation. Maybe that's good enough?
In principle, it is possible to have assets stored outside of the binary: rust-lang/cargo#11683
But this doesn't avoid the problem of the 10MB limit either.

not great...

[hendrikvanantwerpen]

I.e. this is a cost, we don't want to pay when running blackbird tests (or when other ppl depend on this).

This may not be as bad as it could be. I added a print statement to the initialization and it looks like this happens only once for all tests. That was in this crate though, so it could be that in Blackbird it might happens once per crate (if the tokenizers are used directly or indirectly).

I'll try compression and see if that gives us anything.

[hendrikvanantwerpen]

Just compressing the serialized BPE reduces the size, but not enough to get us under the 10MB limit:

	gz (best)	zlib (best)	raw
cl100k	8.4M	8.1M	15M
o200k	17M	17M	31M

I wondered if we could use u16 numbers, but for the o200k automaton, the number of states requires at least 19 bits.

[hendrikvanantwerpen]

@aneubeck The serialization changes in #24 were based on this PR, so most changes here are now the same. Mostly some additional manifest changes.

[aneubeck]

there should be a warning that this is crate is NOT replicating the regex "word splitting" used by openAI.
Therefore, results will differ!

[hendrikvanantwerpen]

I added the warning and also a test that shows an example of the issue.