Plan for Wasming WNFS incrementally

[expede]

A rough plan for how this will break down, and general organizational questions. As an example, I've provided an almost certainly incorrect order of operations below:

Order of Operations

0. Thin interfaces/abstractions (switching between WNFSv1 and WNFSv2 possible)
1. Public WNFSv2
2. Private WNFSv2
3. Switch default to rs-ucan
4. Keystore
5. Blockstore
6. AWAKE & Linking

Dependencies

Consider breaking out libraries as part of this work

• wasm-wnfs
• wasm-ucan
• wasm-awake
• wasm-blockstore
• wasm-keystore

[matheus23]

Sketching out an Interface for wasm-wnfs

Because wasm-wnfs should be invokable from typescript, I'll sketch the interface in typescript:

A first of sketch of the wasm-wnfs core should be just pure functions. No long-lived data structures on the wasm side. If we figure out that this is a bad idea performance-wise, we benchmark the first version and figure out a plan for an interface that allows better optimizations & performance as a second step.

import { CID } from "multiformats"

// e.g. { name: "dag-pb", code: 0x70 } or { name: "raw", code: 0x55 } etc.
interface Codec {
  name: string
  code: number
}

interface BlockStore {
  putBlock(bytes: Uint8Array, codec: Codec): Promise<CID>
  getBlock(cid: CID, signal: AbortSignal): Promise<Uint8Array>
}

export async function nodeLookup(
  fs: CID,
  path: string[],
  { blockStore: BlockStore, signal: AbortSignal }
): Promise<{ remainingPath: string[], entry: Entry }>

type Entry = File | Directory

interface File {
  metadata: Metadata
  content: CID
  previous?: CID
  // ...
}

interface Directory {
  metadata: Metadata
  userland: Record<string, CID>
  previous?: CID
}

I think that single function nodeLookup function is a good first step.
Because it's not a mutation, it doesn't require constructing the right previous link nor does it require figuring out how to write updates to the pretty tree (sorry for these probably unknown terms @appcypher . You can read something about the pretty tree here: https://whitepaper.fission.codes/file-system/partitions/pretty).

[expede]

CHANGELOG: Updated the Issue description to reflect the conversation in Talk

[expede]

@matheus23 even though I'm not the expert on this exact repo, at a quick glance that looks about right for a high level interface 👍

[appcypher]

Shared this on Discord and thought it is relevant here too.

I started work on the Rust WNFS implementation a few days ago and I've been planning how we can make it work both as a regular library that the Rust ecosystem can leverage, as well as make it a good candidate for compiling to WebAssembly. Rust happens to have one of the best toolchains for compiling to lean WebAssembly code and a great JavaScript-based wrapper tool, so this made my work easier although there are still some issues with the current implementation.

I was able to acheive the reusability goal by:

• Moving bindgen and wasm-specific code out into a separate project.
• Avoiding library-specific async features. And if they are really needed, putting them behind a feature flag.
• Using feature flags to choose between a bindgen-based wasm with dependency on JavaScript and a wasm-only build with imports and exports.
• A lot of the dependencies used don't need io std stuff.

I was originally of the impression that asynchronous calls would be a pain to implement in wasm, but I was wrong. Given how Rust represents async code in the memory, the rust-wasm team created a wasm-bindgen-futures library that introduces its own async executor. With it, it is possible to call asynchronous JavaScript functions and vice versa. This is going to make our work a lot easier on the async front.

The Rust WNFS library is currently in a very early stage of development and there are a few issues I'm still trying to figure out.

• Because of the separation of the between Rust-only stuff and wasm-specific code, there is duplication of code.
• The async code crashes sometimes and I haven't figured out why yet.

[appcypher]

@expede What does keystore refer to here?

4. Keystore

• wasm-keystore

[expede]

@appcypher the keystore is where we hold symmetric (e.g. AES) keys, skip ratchet material, and other keys that we need to use for decryption. WNFS has a recursive encryption scheme, but you need to get into that first "entry" node.

[expede]

Related, but getting split out: https://github.com/fission-suite/internal/issues/15