Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Web/WhatWG Streams API Support #98

Open
pojntfx opened this issue Feb 2, 2024 · 2 comments
Open

Web/WhatWG Streams API Support #98

pojntfx opened this issue Feb 2, 2024 · 2 comments

Comments

@pojntfx
Copy link

pojntfx commented Feb 2, 2024

Currently, this library supports the legacy NodeJS streams API (see https://github.com/kriszyp/cbor-x?tab=readme-ov-file#streams), but not the newer Web/WhatWG streams API (see https://developer.mozilla.org/en-US/docs/Web/API/Streams_API). While it is possible on NodeJS to convert from the legacy API to the the new web streams API with Readable.toWeb()/Writable.toWeb(), neither of these APIs are available in the browser, making it hard to use cbor-x's streaming decode/encode support correctly (with support for backpressure etc.) when reading from a Readable stream such as a fetch request body or a WebTransport stream.

Adding support for this new (and arguable simpler to implement) streams API natively would greatly simplify this library's use in the browser.

@MeirionHughes
Copy link

As an aside, if you add 'readable-stream' (node's v18 streams) to your bundler and import from there instead of node:streams, you can use the toWeb, fromWeb helpers in the browser.

@bradisbell
Copy link

bradisbell commented Oct 17, 2024

I worked up a decoder stream, following the Node.js stream code as a reference. I haven't done much testing so I'm not submitting a pull request for now, but let me know if you see any issues...

import { Decoder } from 'cbor-x';

export default class CborDecoderStream {
  #decoder;
  #incompleteBuffer;

  #transform(chunk, controller) {
    if (this.#incompleteBuffer) {
      const newBuffer = this.#incompleteBuffer.length + chunk.length;
      newBuffer.set(this.#incompleteBuffer);
      newBuffer.set(chunk, this.#incompleteBuffer.byteLength);
      chunk = newBuffer;
    }

    let values;
    try {
      values = this.#decoder.decodeMultiple(chunk);
    } catch(e) {
      if (e.incomplete) {
        this.#incompleteBuffer = chunk.slice(e.lastPosition);
        values = e.values;
      } else {
        controller.error(e);
      }
    } finally {
      for (const value of values || []) {
        controller.enqueue(value);
      }
    }
  }

  constructor(opts) {
    opts = opts || {};
    opts.structures = []; // TODO: What is this for??
    this.#decoder = opts.decoder || new Decoder(opts);

    return new TransformStream({
      transform: this.#transform.bind(this)
    })
  }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants