This repository is a fork of hmarr/openai-chat-tokens, modified to reduce the bundle size by replacing unused large objects. The result is that bundle size is ~4 MB, reduced to ~1 MB 🎉.
The primary purpose of this fork is to minimize the size of the JavaScript bundle by replacing large, unused objects with empty objects. This helps improve the performance and loading time of applications that use this package.
The following large objects have been replaced as they were not used in our specific implementation:
gpt2_defaultr50k_base_defaultp50k_base_defaultp50k_edit_defaulto200k_base_default
These objects are defined in the original repository and can significantly increase the bundle size. By replacing them with empty objects, we ensure that the bundle remains lightweight.
You can install the modified package using npm:
$ npm install @mootod/openai-chat-tokensimport { encode, decode } from "@mootod/openai-chat-tokens";
const encoded = encode("Hello, world!");
const decoded = decode(encoded);
console.log("encoded:", encoded);
console.log("decoded:", decoded);We used esbuild to replace the large objects with empty objects in the bundle. For more details, refer to the esbuild.config.js file in the repository.