feat(package): add dictionary compression #276
Conversation
mimi89999
force-pushed
the
compression
branch
from
cb42628
to
c317614
Compare
There was a problem hiding this comment.
Hey, thank you very much for the beginning of the compression!
Maybe you could implement a first implementation from lifthrasiir in #254
This should reduce the dictionary even more 🤔 Or we do it in another PR which is totally fine too!
| @@ -1,3 +1,6 @@ | |||
| import { compressSync, strToU8 } from 'fflate' | |||
| import { encodeBase85 } from '@alttiri/base85' | |||
There was a problem hiding this comment.
I don't really like the idea of using a dependency with almost no downloads. It's kind of a security concern. Maybe we should copy it into this repository instead of depending on it.
I need to think about it 🤔
| const data = Array.isArray(parsed) | ||
| ? `"${parsed.join(',')}".split(',')` | ||
| : JSON.stringify(parsed) | ||
| var code |
There was a problem hiding this comment.
Lint rules, no var allowed. This need to be a let
| var code | ||
| if (Array.isArray(parsed)) { | ||
| const data = parsed.join(',') | ||
| const data_buf = strToU8(data) |
There was a problem hiding this comment.
Lint rule: Need to be camelCase
Well, the sorting would have to be done in the generation scripts where word rankings are still available as the jsons only contain lists without rank score. I think that it's better to do that in a separate pull request. |
Even when including fflate in the bundle, I still got a compression level of 1.6