generate typescript definitions #42

jraymakers · 2024-01-08T18:51:37Z

No description provided.

jraymakers · 2024-01-10T18:25:27Z

package.json

@@ -16,7 +16,7 @@
  },
  "scripts": {
    "install": "node-pre-gyp install --fallback-to-build",
-    "pretest": "node test/support/createdb.js",
+    "xpretest": "node test/support/createdb.js",


This is disabled (by renaming it from "pretest") because it uses the old API.

jraymakers · 2024-01-10T18:26:54Z

generate-wrapper.py

-
-        if isinstance(node.type, pycparser.c_ast.Enum):
-            self.result += f'auto {name}_enum = Napi::Object::New(env);\n'
+            self.cpp_result += f'exports.Set(Napi::String::New(env, "{name}"), duckdb_node::PointerHolder<{name}>::Init(env, "{name}")->Value());\n'


I renamed "result" to "cpp_result", since there is now also a "types_result". I haven't changed the generated C++ at all.

jraymakers · 2024-01-10T18:36:17Z

generate-wrapper.py

+            self.cpp_result += f'exports.Set(Napi::String::New(env, "{name}"), duckdb_node::PointerHolder<{name}>::Init(env, "{name}")->Value());\n'
+            self.types_result += f'export class {name} {{}}\n'
+            self.c_type_to_ts_type[name] = name
+            self.c_type_to_ts_type[f'{name}*'] = name


Each struct type is exposed to JS as a class that is a variant of PointerHolder. This is true for both direct references to the struct type as well as pointers to that type. I'm representing these as separate classes; the only thing that a TS client can do is "new" them and then pass them around.

jraymakers · 2024-01-10T18:37:52Z

generate-wrapper.py

+            # Do these void* types need any corresponding Napi code?
+            self.types_result += f'export class {name} {{}}\n'
+            self.c_type_to_ts_type[name] = name
+            self.c_type_to_ts_type[f'{name}*'] = name


This part handles typedefs like typedef void *duckdb_bind_info;. It's needed because some APIs take or return duckdb_bind_infos.

jraymakers · 2024-01-10T18:38:57Z

generate-wrapper.py

@@ -59,43 +101,60 @@ def visit_FuncDecl(self, node):

       if node.args:
           for p in node.args.params:
-               args.append(typename(p.type))
+               args.append((p.name, typename(p.type)))


I added the name of each arg to the args array, so I can generate the TypeScript function signatures with the correct argument names.

jraymakers · 2024-01-10T18:39:31Z

generate-wrapper.py

           return # ??

       if 'delete_callback' in name:
+           print(f'function not handled: {name}')
+           self.types_result += f'export type {name} = (data: pointer) => void;\n'


This is needed because duckdb_delete_callback_t is used in a couple places.

jraymakers · 2024-01-10T18:41:32Z

generate-wrapper.py


       n_args = len(args)
-       args.append(name)
+
+       fwrap_args = list(map(lambda arg: arg[1], args)) + [name]


Rather than modify args, which I need for the TS generation, I'm creating a copy for the C++ generation. The C++ needs the arg types (not the names), plus the name of the function.

jraymakers · 2024-01-10T18:42:34Z

test.js

@@ -52,7 +52,7 @@ async function test() {

    // we want an incremental AND streaming query result
    const pending_result = new duckdb_native.duckdb_pending_result;
-    await duckdb_native.duckdb_pending_prepared_streaming(prepared_statement, pending_result); // TODO can this fail?
+    duckdb_native.duckdb_pending_prepared_streaming(prepared_statement, pending_result); // TODO can this fail?


This await was unnecessary, because this function is not async.

test.js

jraymakers · 2024-01-10T19:46:09Z

test.js

@@ -4,15 +4,15 @@ const duckdb_native = require('.');
 console.log("DuckDB version:", duckdb_native.duckdb_library_version());

 function convert_validity(vector, n) {
-    const res = new Uint8Array(n).fill(true);
+    const res = Array.from({ length: n }).fill(true);


It seems like this function should return a normal JS array of booleans. This is a more canonical way to do that.

jraymakers · 2024-01-10T19:47:33Z

test.js

    const validity_buf = duckdb_native.copy_buffer(duckdb_native.duckdb_vector_get_validity(vector),
                                                   Math.ceil(n / 64) * 8); // this will be null if all rows are valid
    if (validity_buf == null) {
        return res; // TODO maybe return a singleton so we dont have to allocate?
    }
    const typed_validity_buf = new BigUint64Array(validity_buf.buffer);
    for (let row_idx = 0; row_idx < n; row_idx++) {
-        res[row_idx] = (typed_validity_buf[Math.floor(row_idx / 64)] & (1n << BigInt(row_idx % 64))) > 0;
+        res[row_idx] = (typed_validity_buf[Math.floor(row_idx / 64)] & (BigInt(1) << BigInt(row_idx % 64))) > 0;


Not all JS targets can handle the newer bigint literal syntax.

jraymakers · 2024-01-12T18:28:51Z

src/duckdb_node_generated.cpp

 #include "duckdb.h"
+#include "function_wrappers.hpp"


This change happened automatically when I ran clang format.

jraymakers changed the title ~~generate typescript definitions~~ WIP generate typescript definitions Jan 8, 2024

jraymakers commented Jan 10, 2024

View reviewed changes

test.js Outdated Show resolved Hide resolved

jraymakers changed the title ~~WIP generate typescript definitions~~ generate typescript definitions Jan 10, 2024

jraymakers marked this pull request as ready for review January 10, 2024 18:43

jraymakers force-pushed the jray/refactor branch from 2c2a17e to d4ac3fb Compare January 10, 2024 19:00

jraymakers commented Jan 10, 2024

View reviewed changes

jraymakers added 12 commits January 12, 2024 10:19

generate typescript definitions

4148620

test typescript definitions

7cfbcc6

commit typescript defs

d59ccc1

handle function args

76f5b17

rename result to cpp_result

e214383

remove unnecessary format strings

c2ef285

port test.js to test.ts

d3f0aa5

make test.js and test.ts as close as possible

119bfec

fix indent

d0cef94

update test.ts for test.js changes

586c86e

update test.js and test.ts to be closer togetehr

e88bba3

start of higher-level api

9a9ba59

jraymakers force-pushed the jray/refactor branch from ff0426f to 9a9ba59 Compare January 12, 2024 18:20

jraymakers added 2 commits January 12, 2024 10:26

update generated files

9531c2e

format generated cpp

6d2e4ec

jraymakers commented Jan 12, 2024

View reviewed changes

jraymakers added 23 commits January 12, 2024 15:01

add back duckdb_validity_row_is_valid

3726e3e

deprecate duckdb_vector_ensure_validity_writable

44d2716

start at vector

4c6ac46

add files to index

9eeda9d

change Buffer to UInt8Array

237affc

fix uint64_t* conversion

3f97710

add test for nested list access

62eb2ae

refactor of DuckDBVector to support validity and nested lists

4371681

primitive vector types

7752e16

struct vector

3471539

map vector

3f2d21e

map vector validity

6898b17

union vector

a7e5ac2

test union

ef4e235

destroy config after open

68b48f7

basic api test

bc65ff3

test name change

e165690

prepared statement & pending result

e1bd809

streaming result

b8ab1a4

fix typo in comment

732a9ea

prepared statement test

57a758b

add pending result to index

f0afbb5

global function tests

3abc7cf

hannes merged commit 3d586b3 into duckdb:refactor Jan 30, 2024
3 of 6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

generate typescript definitions #42

generate typescript definitions #42

jraymakers commented Jan 8, 2024

jraymakers Jan 10, 2024

jraymakers Jan 10, 2024

jraymakers Jan 10, 2024

jraymakers Jan 10, 2024

jraymakers Jan 10, 2024

jraymakers Jan 10, 2024

jraymakers Jan 10, 2024

jraymakers Jan 10, 2024

jraymakers Jan 10, 2024

jraymakers Jan 10, 2024

jraymakers Jan 12, 2024

generate typescript definitions #42

generate typescript definitions #42

Conversation

jraymakers commented Jan 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment