could kuzu work with quarto-live leveraging kuzu-wasm?

[Analect]

@DylanShang ... we exchanged messages on the kuzu discussion. I'm interested in your thoughts on the following:

• I have been experimenting with quarto-live and a different pyodide extension for the quarto documentation tool.
• I was hoping to be able to leverage pyodide to give me the ability to render network graphs directly in my documentation, derived from a kuzudb representation of my content, which I generate and store as part of the site's static files. Per this issue, I'm looking to leverage the yWorks/yfiles-jupyter-graphs widget, although I haven't attempted yet to get test if that works in a wasm setting.
• I'm dealing with a small kuzu database, so I don't think I'm impact by some of the reservations that the kuzu developers mentioned pending formal support for kuzu-wasm.
• Your setup in kuzu-lab leverages jupyterlite. Do you think it could also function with a wasm-enabled environment per the quarto-live or coatless-quarto/pyodide tooling?
• I notice over on the ibis-project/tutorials/browser/repl docs site, where they use quarto, that they are able to get a jupyterlite repl running in a quarto context. The underlying code is here. I notice how they leverage await pyodide_js.loadPackage("https://storage.googleapis.com/ibis-wasm-wheels/pyarrow-16.0.0.dev2661%2Bg9bddb87fd-cp311-cp311-emscripten_3_1_46_wasm32.whl") in a similar way to how you use it in your kuzu-use.ipynb demo notebook.
• what function are the files in the kuzu_wasm folder playing?
• is the github-actions deployment script, leveraging the kuzu-wasm npm module central to all this working? I'm running documentation on a self-hosted gitlab for now, so I would probably need to emulate this somehow.
• is it necessary to use netlify or is it enough to host via either github or gitlab? Perhaps if I can get things working in a quarto context, then this should not matter.

Anyway, I appreciate your thoughts on these matters. Thanks. Colum

[DylanShang]

Hi @Analect, I'm very happy to discuss this topic with you.

• I believe implementing this on quarto-live or coatless-quarto/pyodide is very possible.
• Ibis is indeed one of the projects that inspired me.
• Currently, due to limitations with kuzu, cross-origin isolation needs to be enabled in the headers, which GitHub doesn't seem to support.

I'm happy to help you solve this issue, and I'll dive deeper into Quarto. After my exam tomorrow, I'll give you a more detailed explanation.

[Analect]

Great @DylanShang. Good luck with your exam!

[DylanShang]

Hi @Analect
I hope you haven't been waiting too long. I did some work on Quarto and yFiles-Jupyter-Graphs.

is the github-actions deployment script, leveraging the kuzu-wasm npm module central to all this working? I'm running documentation on a self-hosted gitlab for now, so I would probably need to emulate this somehow.

• Yes, or you can also download it in advance. I just want to make sure to use the latest version, but not required in production

Currently, there are a few limitations:

• Kuzu requires threading, which means Cross-Origin Isolation must be enabled during use kuzu-wasm.
• Another issue is that Pyodide does not natively support multi-threaded WebAssembly. The approach I used is inspired from jupylite_duckdb, employing an external loading method. This can answer the question what function are the files in the [kuzu_wasm](https://github.com/DylanShang/kuzu-lab/tree/main/content/kuzu_wasm) folder playing?
  I already raised an issue#4107 with Kuzu about threading, but a short-term resolution seems unlikely.

The good news is that I should have successfully integrated Quarto, yFiles-Jupyter-Graphs, and Kuzu. I'll publish code in few days and mention you. However, there are some limitations:

• yFiles-Jupyter-Graphs rely on Jupyter-widget. My experiment at quarto-live worked but failed to plot in pyodide extension, my code is

import micropip as mp
from IPython.display import display
await mp.install("yfiles-jupyter-graphs")
from networkx import florentine_families_graph
from yfiles_jupyter_graphs import GraphWidget
w = GraphWidget(graph = florentine_families_graph())
display(w)

• Due to the Cross-Origin Isolation issue, you might need to host Kuzu-lab on your own website. A potential solution can be found at coi-serviceworker. I haven't try yet, and I will do that in few days.
• Loading yFiles-Jupyter-Graphs in JupyterLite failed initially, but I managed to fix it #55. However, another issue arose: using yFiles-Jupyter-Graphs under your own domain requires authorization. I suggest reaching out to them to obtain the necessary permissions. I did it here.

At last, I will keep you informed of any developments regarding quarto in this issue

[Analect]

@DylanShang ... great progress. Thanks for the update. I hadn't realised that there was that limitation on the yFiles widget. It's hard to find good visualisation solutions for graphs that don't require a running server, per kuzu-explorer or Neo4j Browser. Maybe something like pyvis can work in a wasm context ... was looking here, here and here for inspiration.

[DylanShang]

@Analect At present, pyvis does seem to be the best option, and it can also run on WebAssembly. Could you share your use case and requirements? I can look for alternatives if there is a better fit.

[Analect]

@DylanShang ... sorry for the delay in getting back to you.

I've created a partly worked-through example use-case here: https://analect.github.io/quarto-graph-experiments/docs/demo/kuzu-use.html

There are other resources on this documentation site too ... but the idea is get a kuzu / pyvis example working the in the kuzu-use.qmd

The quarto doc site, as part of pre-render scripts, scrapes metadata (the yaml part at the top of each qmd and ipynb doc), pushes these to csv files in the data directory and then a second script loads these as a kuzu db to data/kuzu-test.

While there are various things I want to do with this meta-data, one of the basic use-cases I have is to be able to render a quarto document and use the graph model to give context to related content, akin to functionality in obsidian, per this example.

So, in my use case, I would look to run a basic cypher query that brings back the current document and maybe display other documents that have common categories. I will play around with this ... but the main need I have is to get kuzu, working with something pyvis, while leveraging either pyodide or quarto-live extensions of quarto.

In the mocked-up doc site, I have included other examples from Github where the authors are pushing the boundaries in terms of adding graph visualisations to documents. In most cases, these are getting created by either R or Python code-blocks that get executed at render-time ... rather than leveraging wasm ... which is what I'm trying to do with my use-case.

Does this make sense?

[DylanShang]

Hi, @Analect
I'm working on using PyVis with Quarto-live and the Pyodide extension, but the rendering fails, and the interface remains blank. You can tell me if you get success in either of them.
And it works fine in Jupyter's REPL.I'm wondering if you would consider embedding the REPL as an iframe into your website, similar to how Ibis was used. Recently, I’ve been packaging kuzu_wasm into a Python wheel to reduce the amount of code needed during usage. In a few days, I’ll be releasing a qmd document utilizing the Jupyter REPL for your reference.

[Analect]

Hi @DylanShang.
Yes, I've experimented with embedding jupyterlite within my docs, per the embedded jupyterlab below. This works. But it's still my preference to get things working directly within either a *.qmd or *.ipynb file, for quarto to render on my behalf.

Reverting back to the mock-up site I set up, as you said, neither of the approaches in cell 1 (akin to what Ibis are doing here) or cell 2 (leveraging micropip) are working per the rendered qmd at https://analect.github.io/quarto-graph-experiments/docs/demo/kuzu-use.html. This is the underlying code.

I suppose the first approach might work if pyodide_js.loadPackage() might be leveraged to discover your kuzu_wasm folder. Am I right in thinking that wasm isn't able to load from local files. I was looking at this issue. I suppose I don't have enough knowledge around wasm and how to appropriately package *.whl files, as yet.

[DylanShang]

Hi @Analect
Your idea is correct. I think using a remote loading method for the wheel is a better approach.

Additionally, I’ve deployed the JupyterLite console version with Quarto; The underlying code is here. If you encounter any issues, try clearing all caches for the site first. And sorry for the misunderstanding caused by my explanation. What I meant by a method similar to Ibis was referring to this style of presentation.

Compare with Quarto-live, I think Quarto-live does have a better presentation for blog.

I think the main issue right now is that PyVis cannot render on Quarto-live(The code runs without errors, but the output appears blank) , and I suspect this might be related to issues #7 and #44.

This is the code I tried failed in Quarto-live but successfully in jupyterlite

import micropip
await micropip.install("pyvis")
import pandas as pd
import numpy as np
import networkx as nx
from IPython.display import display, HTML
import matplotlib.pyplot as plt

data = [{'Circuit': 'html', 'Description':1, 'Duration':10.2, 'Source':'Westchester', 'Destination':'Davie', 'Picklist':1000, 'Postlist':50000.2}, 
        {'Circuit': 'html', 'Description':2, 'Duration':12.1, 'Source':'Westchester', 'Destination':'Davie', 'Picklist':3000, 'Postlist':40000.1},
        {'Circuit': 'html', 'Description':3, 'Duration':11.3, 'Source':'Westchester', 'Destination':'Davie', 'Picklist':7000, 'Postlist':50000.2}, 
        {'Circuit': 'html', 'Description':3, 'Duration':8.1, 'Source':'West', 'Destination':'San Bernardino', 'Picklist':3000, 'Postlist':40000.0},
        {'Circuit': '.net', 'Description':4, 'Duration':6.2, 'Source':'Queens', 'Destination':'San Bernardino', 'Picklist':5000, 'Postlist':6000.1}, 
        {'Circuit': '.net', 'Description':3, 'Duration':20.1, 'Source':'Queens', 'Destination':'Los Angeles', 'Picklist':5000, 'Postlist':4000.1},
        {'Circuit': '.net', 'Description':2, 'Duration':15.5, 'Source':'Brooklyn', 'Destination':'San Francisco', 'Picklist':5000, 'Postlist':9000.3},
        {'Circuit': '.net', 'Description':4, 'Duration':7.7, 'Source':'Brooklyn', 'Destination':'Davie', 'Picklist':6000, 'Postlist':10000},
        {'Circuit': '.net', 'Description':4, 'Duration':7.7, 'Source':'Los Angeles', 'Destination':'Westchester', 'Picklist':6000, 'Postlist':10000},
        {'Circuit': '.net', 'Description':4, 'Duration':7.7, 'Source':'San Bernardino', 'Destination':'Westchester', 'Picklist':6000, 'Postlist':10000}]
df = pd.DataFrame(data)

G = nx.from_pandas_edgelist(df, source='Source', target='Destination', edge_attr='Picklist')

from pyvis.network import Network
g = Network(notebook=True, cdn_resources='remote')
g.from_nx(G)
html = g.generate_html(notebook=True)
HTML(f'<div style="height:{g.height}">{html}</div>', metadata={"isolated": True})

Perhaps we should track this issue or use a method like quarto-example if it suits. Alternatively, we could try to find another visualization library that works properly on Quarto-live.

[DylanShang]

Or maybe you will like the form of embedding the notebook into your website like this, but I think it's not good for SEO

Anyway, when you decide on the way of presentation, we can proceed further.

[Analect]

@DylanShang . Thanks alot for your efforts. By any chance, does leverage iPython display work with materialising the PyVis output? This solution was suggested to get things working in a Colab context, but perhaps it works for quarto-live.

from IPython.core.display import display, HTML

g.show('network.html')
display(HTML('network.html'))

[DylanShang]

Sorry, I got failed. The result is still empty. You can also try it here with above code. It seems like it has be blocked. Maybe we could post a issue first? And I need few days to know why.

[Analect]

Tried putting all your code into a single pyodide cell ... and get back ReferenceError: SharedArrayBuffer is not defined, which seems to be something to do with cross-origin stuff!

[DylanShang]

You can enable it by using coi-serviceworker or host in Netlify and use this config in root

And you may need to host kuzu js/wasm in your own domain and change the URL in kuzu_wasm init fucniton.

[DylanShang]

Hi, @Analect
This morning I did some effort on pyvis, I think I've fixed the problem.
I'll publish a use case later.

[Analect]

@DylanShang ... great. I'll admit, this cross-origin stuff bamboozles me!

[DylanShang]

Hi, @Analect

Here is demo, and the underlying code

Make sure:

1. Download kuzu-wasm npm package in your project, so that you can get access {rootUrl}/package/dist/kuzu.js, and you can do it by following commands

npm view @kuzu/kuzu-wasm dist.tarball | xargs curl -o package.tgz
tar -xz -C dist -f package.tgz && rm package.tgz

2. You can also host pyarrow wheel file in your project to speed up access. Tip: we do not have to host it in the next version of pyodide. See this.
3. Enable cross-origin isolation, see here

Let me know if you have any issue!

[Analect]

@DylanShang ... thanks alot. I will try this. A quick question - with these changes, is it still necessary to host this on netlify or can it work on github pages with this coi-serviceworker you mention? Is it enough to follow the pattern you use here for installing kuzu-wasm, as it seems to publish its package to npm too.

Just a broader question, what is involved in generating one of these pyarrow wheel files. Is that the one coming from here? Is this necessary, since pyarrow is now a dependency of pandas? Will this enable import pyarrow and for that python package to work in a wasm context too?

[DylanShang]

I know this sounds paradoxical, but Cross-Origin Isolation complicates our deployment process. This is because multi-threaded WebAssembly depends on workers, and workers require SharedArrayBuffer for communication. However, SharedArrayBuffer necessitates Cross-Origin Isolation. Enabling Cross-Origin Isolation prevents us from requesting JavaScript files from non-same-origin sites, so for now, my solution is to host kuzu-wasm on our own site.

• Using coi-serviceworker, I think we can even host it on GitHub. Do you need a demonstration？

• Yes, You can refer to the deploy.yaml for usage instructions. what you mainly need is coi-serviceworker.min.js

• As for pyarrow, yes, I currently deploy it in kuzu-lab by downloading from there since Pyodide doesn't yet support it. However, maybe in a few weeks, you just need to use by import pyarrow directly.

[Analect]

Do you need a demonstration？

I think so!
I'm struggling to get things working on my end.
I tried to emulate your deploy.yml, except I publish to github-pages and not netlify. I couldn't get this to work, so I then tried including the various js files in a dist/package/dist/ folder in my repo (trying to emulate how these libraries get expanded as part of the deploy script) and then including the dist as a resource in _quarto.yml.

The upshot is that I end up with the various files I think I require in gh-pages/dist/package, but somehow, per the image below, they are not discoverable.

I'm also not clear if I should be adding this directly into my *.qmd file or not.

[DylanShang]

@Analect Don't worry it, I'll take a detailed look at your project

[DylanShang]

Hi, @Analect
A available case is here, and the repo

I deleted a lot of redundant files. You can use it by replacing https://dylanshang.github.io with https://analect.github.io

Good luck!

[Analect]

@DylanShang ... thanks alot for your help in trouble-shooting that. I emulated your changes and got it to work my end too. In terms of an automated way of running this in future, to reflect any updated to your kuzu_wasm tooling, would you recommend that I try to get this functioning in a github-actions script or is it better to build up-front in the repo using a Makefile and then pushing that github, which mimics what we (you) have had to do to get this working.

Did you plan to roll kuzu_wasm into its own whl, which would get added to the requirements.txt?
For now, it's necessary to host the pyarrow-17.0.0-cp312-cp312-pyodide_2024_0_wasm32.whl locally and then point back to the hosted version?
In terms of the files __init__.py and utils.py that I moved to the unused folder below. They presumably no longer serve a purpose?

├── package
│   ├── kuzu-wasm
│   │   ├── kuzu.js
│   │   ├── kuzu.js.map
│   │   ├── kuzu-wasm.js
│   │   ├── kuzu-wasm.wasm
│   │   └── kuzu-wasm.worker.mjs
│   ├── unused
│   │   ├── __init__.py
│   │   └── utils.py
│   └── wheels
│       └── pyarrow-17.0.0-cp312-cp312-pyodide_2024_0_wasm32.whl

I will experiment around with the pyvis / kuzu combination as a means for exploring a documentation site. By any chance, have you seen any working examples of graphs that have 'clickable' events? Where clicking on the node would take you to a different URL within a documentation site, for example. I notice this PR within pyvis, but it seems that the project is no longer actively maintained.

Thanks again for your help with this. It's been instructive.

[DylanShang]

Hi @Analect,

1. My suggestion is to keep things as they are for now, as I'm planning significant changes in the next version. However, this will likely not improve your project, as kuzu_wasm has already served its primary purpose.

2. Pyarrow will be included in the next Pyodide release, so we won't need to host it separately anymore.

3. The kuzu_wasm Python package has been bundled into its own .whl file, which means init.py and utils.py will no longer be necessary.

4. I'm a bit confused about your requirements. If you're simply looking to achieve something like Obsidian's functionality, you could pre-render it into static graph. I think the current approach using kuzu-wasm/pyvis is more suited for dynamic graph research/study. It’s possible that you might be overcomplicating things. This assessment is based on my current understanding of your project, and I'd be happy to provide more tailored advice if you can share additional details about your needs.

[Analect]

It’s possible that you might be overcomplicating things. This assessment is based on my current understanding of your project, and I'd be happy to provide more tailored advice if you can share additional details about your needs.

@DylanShang ... you are right. I'm probably not being clear. Setting aside the obsidian use-case, I am actually more interested in the dynamic graph generation via quarto-live cells, which would allow one to explore many different cypher queries on a data-set that gets dynamically updated, when a documentation site renders, for instance.

Per this issue, the devs at quarto-live were able to help me fix accessing a data folder from a cell. Per this updated kuzu-quarto-live, where cells are set to be run manually, I'm able to access the data folder as well as a small kuzu database at data/kuzu-test. I download that same database locally and interfaced with a local Kuzu CLI to confirm that it functions and contains a small amount of data.

However, per the third cell, I'm unable to get the kuzu-wasm working to connect to that pre-existing on-disk data/kuzu-test database. Is that a known limitation? Is it intended to just work with in-memory variants? Do you see anything obvious that I'm missing here?

[DylanShang]

Hi, @Analect
The current solution is download data files from server like this and import it when kuzu get ready.

kuzu.FS.writeFile("/user.csv",await (await js.fetch("https://raw.githubusercontent.com/kuzudb/kuzu/master/dataset/demo-db/csv/user.csv")).text())

This is because the file systems in kuzu-wasm and pyodide are separate now.
What's more, I will develop an interface to read directly from a local database on pyodide in the following days. Thanks for your idea. I'll let you know when it's updated.

[Analect]

This is because the file systems in kuzu-wasm and pyodide are separate now.

Ah OK @DylanShang . Now it's clear. The other thing that I was toying with was to ATTACH to the database hosted on Github. The below works , however, per this issue, you need to run it against an on-disk instance of a kuzudb. It fails if you try it with an in-memory one. So if the code snippet below were called remote-db-github.cypher, you would run kuzu ./test-github < remote-db-github.cypher. However, as I understand, ATTACH is only good for read-only, so your proposed solution of accessing a read-write database in the pyodide VFS seems like the right way to go.

INSTALL httpfs;
LOAD EXTENSION httpfs;
CALL HTTP_CACHE_FILE=TRUE;
ATTACH "https://raw.githubusercontent.com/analect/quarto-graph-experiments/gh-pages/data/kuzu-test" AS meta (dbtype kuzu);
CALL SHOW_ATTACHED_DATABASES() RETURN *;
MATCH (a:Document)-[f:authored_by]->(b:Author) RETURN a.Title,f,b.Author;
DETACH meta;

In terms of your interirm solution above ...

kuzu.FS.writeFile("/user.csv",await (await js.fetch("https://raw.githubusercontent.com/kuzudb/kuzu/master/dataset/demo-db/csv/user.csv")).text())

... can this be modified to handle a whole folder (rather than a single file, as required by the kuzu database specification) ... or does one need to recurse through the folder to perform a kuzu.FS.writeFile for each component file manually?

[DylanShang]

Sorry about that, kuzu-wasm is still at version 0.4.2, and the httpfs extension is not supported in this version.

I've been working all day on resolving file transfer issues between Pyodide and kuzu-wasm, but I seem to be stuck at the moment, and it might take some time to solve this.

Therefore, I strongly recommend using the method I mentioned earlier. You can replicate what you've done in load_data_kuzu.py within quarto-live.

#create schema 
await conn.execute("CREATE NODE TABLE Category(Category STRING,PRIMARY KEY (Category))")
...
#import csv 
kuzu.FS.writeFile("/core_meta-data.csv", await (await js.fetch("https://github.com/Analect/quarto-graph-experiments/raw/gh-pages/data/core_meta-data.csv")).text())
...

This might be quite cumbersome, but it's the only solution I have at the moment. I will explore further to support more import methods.

[DylanShang]

I think I have to upgrade the version of kuzu within kuzu-wasm to achieve backward compatibility with kuzu's storage filesystem so we can use different versions of kuzu files directly. But this might take a long time.

[Analect]

@DylanShang. I noticed you were able to get the latest kuzu version working at unswdb/kuzu-wasm. Does that mean there is now a mechanism to pull in a pre-formed kuzudb rather than having to recreate it within the kuzu VFS from files? For you kuzu shell , is it possible to get this into a console like ibis have done with jupyterlite. Thanks for your efforts.

[DylanShang]

Hi, @Analect . Thank you for your continued interest.
I recently upgraded the version of kuzu-wasm to 0.6, which means it can now load database files generated by all previous versions theoretically. I'm currently working on a GitHub Action that will allow you to pre-package the necessary files into the wasm module. This means you'll be able to include your own database files directly within the wasm, eliminating the need to fetch them from the network each time.
Additionally, I'm working on a few other tasks, and you can expect a release within the next two weeks.

For you kuzu shell , is it possible to get this into a console like ibis have done with jupyterlite

Do you mean this?

[Analect]

@DylanShang ... thanks for your great progress on this. I look forward to working with your solution.

Ref. the REPL, you are right, I had forgotten that implementation, per 2 below. I think it still leverages your old approach, but I expect that will alter as your new mechanism gets implemented. I note how Ibis use %clear to hide any setup boiler-plate per here, which is useful. As you know, finding a good network-visualisation solution that's intuitive in the REPL is still a challenge.

I can see the attraction of the jupyterlite REPL, but directly accessing a kuzu shell, per 3 above, particularly if you could pre-wire it to an on-disk kuzudb (as per your work above) would be handy ... in order to be able to directly run cypher queries without any of the boiler-plate set-up required for the REPL. Is it straight-forward to be able to drop the shell into a *.qmd file, as per the REPL, or is that somehow more involved?

[DylanShang]

@Analect Do you mean embed kuzu-shell into a qmd file?
The kuzu-shell is a standalone application built with React, which may not be easily embedded directly. The only feasible way to display it is via an HTML iframe, but this requires hosting it on same domain. However, I’m not inclined towards this approach as it could add unnecessary complexity and bloat to the project. Since QMD natively supports Python code, using conn.execute("cypher") might be the most suitable and streamlined method for integrating with QMD.

[DylanShang]

Hi, @Analect
A available case is here, code is here.

I made a docker image so everyone could embed their KuzuDB files or other data into the wasm. Usage is here. In this example, all the files in the data folder will be mapped into the root directory of wasm'VFS. A new wasm package will output to package/kuzu-wasm in this example.

Also, I found out that kuzu's db file system is not backward compatible, so specific kuzu-wasm can only read the db files created by specific kuzu python. Current kuzu-wasm use the latest storage_version (v32). So you have to use the kuzu python with v32 storage_version. The way to show storage_version is here.

Another thing is according to this, I got a new way to add the hyperlink in pyvis.

Feel free to ask me if there are other questions!