Request for jq and json2csv

[hhhuff]

I would love to see if someone could add jq and json2csv as I have a repeated workflow that include these tools. Thank you.

[fprimex]

I have jq working in the simulator and am trying to finish up getting it added. I'm not familiar with json2csv. Do you have a link to it?

[hhhuff]

Yes Sir, jehiah/json2csv: command line tool to convert json to csv (github.com)<https://github.com/jehiah/json2csv> From: Brent Woodruff ***@***.***> Date: Friday, June 3, 2022 at 4:39 PM To: holzschu/ios_system ***@***.***> Cc: hhhuff ***@***.***>, Author ***@***.***> Subject: Re: [holzschu/ios_system] Request for jq and json2csv (Issue #131) I have jq working in the simulator and am trying to finish up getting it added. I'm not familiar with json2csv. Do you have a link to it? — Reply to this email directly, view it on GitHub<#131 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AV2A4PNJCEVD5KNTBQ4CYTLVNJ3QFANCNFSM5UDXACGQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

…

-- ******************************************************************* This message was sent from RiskIQ, and is intended only for the designated recipient(s). It may contain confidential or proprietary information and may be subject to confidentiality protections. If you are not a designated recipient, you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you. *******************************************************************

[fprimex]

I had a quick look and the difficulty here is that the program is written in Go. This language toolset is not yet working with ios_system. I compiled it and it produces neither output nor files.

GOOS=js GOARCH=wasm go build -o json2csv.wasm

Go can natively produce both wasm and ios/arm64 formats, however the wasm SDK is not the one needed for to produce compatibly ios_system output.

The ios/arm64 output would work, but Go does not support outputting dynamic, shared libraries on ios/arm64. I tried combinations of -buildmode=shared, buildmode=c-shared, and -linkshared but none were accepted.

Honestly, your best bet for getting json2csv is to port it to python or jq. jq has an @csv output target. I would like to see Go work with ios_system at some point, but I feel like that is probably a long ways off.

[fprimex]

I would be interested in decompiling the resulting Go wasm, replacing the needed SDK calls, and then reassembling it. That would enable Go programs in ios_system. I'll try that later.

[fprimex]

Used wabt to convert the wasm to c. It was over 32MB and 1M lines long 😅. It must have the entire Go stdlib in it. I tried compiling it, but killed it when it had consumed 18GB of RAM on my system.

[holzschu]

I've been thinking. WASM should be portable (it is designed for that). The only potentially non-portable parts are when it tries to access the file system, which is usually encapsulated into JavaScript commands. What kind of errors do you get with json2csv.wasm?

[fprimex]

IIRC it just doesn't work. No output, no errors, no files.

[holzschu]

That doesn't make a lot of clues, I'll grant you that. But normally, a webAssembly command that does not execute produces an error, so "no errors" could be a positive sign. Since webAssembly does not have access to the file system, I would try with stdin and stdout (json2csv < input.json > output.csv).

[fprimex]

This similarly produces no output.

[fprimex]

Go wasm does not provide wasi and therefore cannot be used in the context ios_system / a-shell wants. The tinygo provider, however, does and compiling with it works fine. Unfortunately, tinygo does not support everything that Go does, and the resulting wasm "panics" with a reflect.NumMethod not implemented error. This is apparently common for tinygo programs using encoding/json.

I'm looking at one other option and then I think I'm out of ideas. The good news is that using jq to do this would be relatively simple.

[holzschu]

I haven't looked at the Go documentation, but if their WebAssembly version comes with a JavaScript container, maybe we can run this JS container rather than ours? jsc lets you run JavaScript commands, and the code a-Shell uses to run WebAssembly is in wasm.js. If it works, we will have to design a way to switch between JS containers depending on the WebAssembly flavour.

[fprimex]

Here is the working wasm file for json2csv to place in Documents/bin. GH made me zip it so I could upload it 🙄 , so it'll need to be unzipped as well.

json2csv.wasm.zip

[holzschu]

Thanks for compiling this command.
I've made it available on https://github.com/holzschu/a-Shell-commands/ but when I test, I get blank output every time. I might be using it wrong, though.

[fprimex]

I think that version is old and broken. I have attached a tar.gz with the wasm and some tests that can be run that are from the project's acceptance tests and docs. See the tests.txt file for what to run and the expected output (there will be some newline representation differences because of how I copied the tests over most likely).

json2csv_wasm.tar.gz

[holzschu]

It works with your tests, and it works with a metadata.json file that came with a Python package, but with this colors.json, it only produces a lot of blank lines.
colors.json.zip

I'm not knowledgeable enough to know whether this is normal or not.

[fprimex]

colors.json, and basically any single JSON value, isn't going to work for this.

Converts a stream of newline separated json data to csv format.

colors.json is a single JSON value that is on multiple lines. json2csv wants many JSON values, one on each line. So I think the blank lines are a result of it reading each line, failing to do anything with them, but continuing. I figured there would be some stderr since there are some messages in there, but 🤷

[fprimex]

Also, I pushed my entire working directory for the jq port in case you want to see the changes and add it before I get the "build recipe" script done.

https://github.com/fprimex/jq

The branch is iOS, which I set to the default.

[holzschu]

Thanks a lot for cross-compiling jq. I'm including it in the next TestFlight version.

One point I haven't mentionned so far: ios_system does not clear memory or re-initialize variables when you leave a command (because there is no fork/exec, so the command is still in the process memory space, so variables keep their values). Depending on how the command is coded, this can result in memory leaks or options that keep being set on subsequent runs (you run "command -v", so this one is verbose, and then "command", but it is still verbose because the verbose flag is still set from the previous run). Again, that depends on how the command has been coded (some reinitialize all variables and free everything, others don't). I tried looking at the source of jq, but I couldn't find where it parses options.

Update: it seems to hold in my tests. I had a lot of issues with awk, who has a similar syntax. I guess the programmation style of jq is more modern and robust.

[holzschu]

Hi,
the current TestFlight version for a-Shell mini now has jq, thanks to @fprimex: https://testflight.apple.com/join/REdHww5C

You can also install json2csv, which has been compiled to WebAssembly, using: curl -OL https://github.com/holzschu/a-Shell-commands/releases/download/0.1/json2csv, followed by mv json2csv ~/Documents/bin.