Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[This PR was previously submitted to the main duckdb repo.]
Add plumbing to expose the Tokenize method through the Node client API.
The signature of this method follows the corresponding method from duckdb-wasm. It returns an object containing two arrays of numbers, instead of an array of objects, each with two number properties (as the underlying C++ API does), presumably to reduce the number of JS objects that need to be created.
I opted to call the Tokenize method synchronously, since it's lightweight and doesn't require any database resources; it's a static method of the parser. If there's a reason to move the call to the worker thread, let me know.
In addition to exposing the method itself, the
TokenType
enum is also exposed. This requires some code on the C++ side in order to add the runtime enum object to the NodeDuckDB Addon, in addition to the type definition on the TypeScript side.A basic unit test suite is included.