dash: pipes are not working - RuntimeError: unreachable

[milahu]

this will crash the shell at https://cowasm.sh/

seq 1 10 | head -n 1

js console

658.bundle.js:2 /usr/bin/sh RuntimeError: unreachable
    at growjobtab (wasm://wasm/00521052)
    at makejob (wasm://wasm/00521052)
    at evalpipe (wasm://wasm/00521052)
    at evaltree (wasm://wasm/00521052)
    at cmdloop (wasm://wasm/00521052)
    at main (wasm://wasm/00521052)
    at e.default.run (658.bundle.js:2:52854)
    at Object.cowasm_vforkexec (658.bundle.js:2:4329)
    at n.<computed> (658.bundle.js:2:6410)
    at /aa318ea0a6040fd50963.wasm

[williamstein]

Thanks for adding this to the todo list and trying out the cowasm shell. I have indeed not implemented dash pipes yet at all. It requires running a subprocess (all in the same WebAssembly VM) and capturing the output, which is something that probably isn't too hard to figure out and will be exciting. But it's not done.

Any other thoughts about cowasm?

[martin12333]

Any other thoughts about cowasm?

I have several thoughts. My first thought:

one of my dream projects is something like: fragments of Ubuntu compiled to wasm

But I will have to learn a lot.

I am glad, that I was able to install cowasm dash and utils locally.
(my experiments are at
https://github.com/martin12333/marti-onedrive/blob/main2/cowasm/
)

[williamstein]

fragments of Ubuntu compiled to wasm

Yes, that's basically one way to think of cowasm, except with Ubuntu maybe replaced by FreeBSD. The FreeBSD utilities are mostly much smaller and easier to hack on that corresponding GNU tools, so I was able to build a ton of them for WASM this way, and you can see them all in the shell.

Anyway, welcome to the project!

[milahu]

It requires running a subprocess (all in the same WebAssembly VM) and capturing the output, which is something that probably isn't too hard to figure out and will be exciting.

note that pipes are "incremental", so in theory, they can work on infinite data streams

examples ...

seq 1 100000000 | head -n 1 should write 1\n from seq to head
and head should stop seq after the first line

cat /dev/random | head -c 1 | xxd -p should read only 1 byte from /dev/random

cat data.txt | sort --random-sort | head -n 1
should read only some lines of data.txt

there is also a difference between line-buffered and unbuffered ...
line-buffered is useful when working with line-based text files

devil in the details ; )

[williamstein]

note that pipes are "incremental", so in theory, they can work on infinite data streams

I realize that. Unfortunately it's impossible to support that using the model I'm implementing for this with the current Wasm spec (of one single wasm instance and general wasm code). There's just no way to support multiple processes running at once in general. I might also implement a second much more complicated and less efficient subprocess execution model someday that supports that.

[williamstein]

There are a lot of tradeoffs with speed versus functionality. One extreme for situations where speed is not important, there is https://copy.sh/v86/?profile=archlinux, which is a full Linux install running under WASM via an x86 emulator. But it's an order of magnitude (or more) slower at everything, which is not acceptable for me.

[milahu]

speed versus functionality

both ; )

I might also implement a second much more complicated and less efficient subprocess execution model someday that supports that.

a process scheduler ...

sounds like i should continue https://github.com/milahu/shelljs-async
where pipes are "yield buffered" https://github.com/milahu/shelljs-async/blob/main/src/lib/pipe.js
i just need to rewrite everything as async generators ...
see also https://github.com/miketalbot/js-coroutines

[williamstein]

a process scheduler...

There was something that did this before WebAssembly -- https://browsix.org/ -- but it was like 100x slower at least, so obviously nobody used it. Of course with WASM and SharedArrayBuffers/Atomics, it is possible to do something that is fast. That said, sharing state, e.g, of the filesystem, environment, etc., becomes much more complicated and slower. Right now, memfs lives only in memory in one single WebWorker, whereas a multi-process approach like Browsix had requires having multiple WebWorkers, and the filesystem becomes much more complicated to share between them. But it's possible.

[williamstein]

Thanks for the links!

[martin12333]

(related: wasmerio/ate#10 , demo is at https://wasmer.sh/ , wasmerio/webassembly.sh#98 (comment) )

[williamstein]

Neat - I didn't know they actually ported dash and were using that in wasmer.sh too. I wonder if they are solving all the same problems on my todo list already or what approach they have taken... E.g., dash uses setjmp/longjmp, so I've been rewriting it to not do that.

[williamstein]

I suspect it is dash mainly because of

[martin12333]

Neat - I didn't know they actually ported dash and were using that in wasmer.sh too

(if I understand correctly, dash is only in the https://wasmer.sh (the recent website ... i mean , dash is not in the webassembly.sh )

( in the webassembly.sh (the old website), there is probably a js-written, old, mini-shell )

[williamstein]

(if I understand correctly, dash is only in the https://wasmer.sh/ (the recent website ... i mean , dash is not in the webassembly.sh )

That makes sense, because I don't think I have saw wasmer.sh before. In any case, it's pretty cool that I had the idea to use dash to make a WebAssembly shell at the same time as somebody else, and we both did it... :-).