Getting "EXPECTED value GOT EOF" for any kind of JSON in Cygwin

[philippgille]

I'm using JSON.sh in a Cygwin environment and I'm getting an error when trying to use it for any kind of JSON:

EXPECTED value GOT EOF

• Tried with your example: curl registry.npmjs.org/express | ./JSON.sh | egrep '\["versions","[^"]*"\]'
• Also tried with: echo "{\"foo\":\"bar\"}" | ./JSON.sh

Maybe related to #26? Or maybe related to CRLF vs. LF on Windows vs. Linux (note I'm using Cygwin on Windows)?

[dominictarr]

\ does not work as an escape in bash as you think it would. try echo '{"foo":"bar"}'
I wouldn't be surprised if it was CRLF, although I am a little surprised that no one has complained about this before.

[philippgille]

Thanks for the quick reply. echo '{"foo":"bar"}' | ./JSON.sh doesn't work as well.

I'm pretty sure now it's a CRLF problem, because today I used JSON.sh on a Linux machine with the same request I first tried on my Windows machine with Cygwin and it worked fine.

To reproduce: I didn't install Cygwin, instead I use the free portable version of MobaXterm, an SSH client on steroids. Right after launching you're in a Cygwin terminal. Then:

apt-cyg install curl
curl -L https://raw.githubusercontent.com/dominictarr/JSON.sh/master/JSON.sh > JSON.sh
chmod 755 JSON.sh
curl http://echo.jsontest.com/foo/bar | ./JSON.sh -l

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    15    0    15    0     0     77      0 --:--:-- --:--:-- --:--:--    81
EXPECTED value GOT EOF

Update:
Wait a second, in {"foo":"bar"} there's no CRLF. Maybe my versions of gawk or egrep behave differently from the usual ones, so the json doesn't get tokenized correctly and ''|[!0-9] can't match? My tools seem to be from: BusyBox v1.22.1 (2014-09-26 07:33:17 CEST) multi-call binary.

[dominictarr]

can you run the tests? which of the tests fail? awk/grep could be the problem. Possibly, we could change those bits to use options that are more widely supported. You could debug this by adding.

do the parts work? do . JSON.sh and then tokenize < data.json that should give you a json token perline. Post the output.

[philippgille]

Tests:

/drives/e/_temp/JSON.sh-master/all-tests.sh
TEST: test/invalid-test.sh
1..22
ok 1 - invalid/bad_unicode_sequence.json was rejected
# EXPECTED value GOT EOF
ok 2 - invalid/bareword.json was rejected
# EXPECTED value GOT EOF
ok 3 - invalid/bracket_key.json was rejected
# EXPECTED value GOT EOF
ok 4 - invalid/colon.json was rejected
# EXPECTED value GOT EOF
ok 5 - invalid/colon_obj.json was rejected
# EXPECTED value GOT EOF
ok 6 - invalid/comma.json was rejected
# EXPECTED value GOT EOF
ok 7 - invalid/comma_obj.json was rejected
# EXPECTED value GOT EOF
ok 8 - invalid/control_char_in_string.json was rejected
# EXPECTED value GOT EOF
ok 9 - invalid/decimal_point.json was rejected
# EXPECTED value GOT EOF
ok 10 - invalid/empty.json was rejected
# EXPECTED value GOT EOF
ok 11 - invalid/false_key.json was rejected
# EXPECTED value GOT EOF
ok 12 - invalid/null_key.json was rejected
# EXPECTED value GOT EOF
ok 13 - invalid/number_key.json was rejected
# EXPECTED value GOT EOF
ok 14 - invalid/trailing_array_comma.json was rejected
# EXPECTED value GOT EOF
ok 15 - invalid/trailing_garbage.json was rejected
# EXPECTED value GOT EOF
ok 16 - invalid/trailing_object_comma.json was rejected
# EXPECTED value GOT EOF
ok 17 - invalid/true_key.json was rejected
# EXPECTED value GOT EOF
ok 18 - invalid/unclosed_array.json was rejected
# EXPECTED value GOT EOF
ok 19 - invalid/unclosed_object.json was rejected
# EXPECTED value GOT EOF
ok 20 - invalid/unclosed_string.json was rejected
# EXPECTED value GOT EOF
ok 21 - invalid/weird.json was rejected
# EXPECTED value GOT EOF
ok 22 - invalid/weird_key.json was rejected
# EXPECTED value GOT EOF
0 test(s) failed
OK: ---- test/invalid-test.sh
TEST: test/parse-test.sh
1..4
EXPECTED value GOT EOF
not ok 1 - "oooo"
EXPECTED value GOT EOF
not ok 2 - [true, 1, [0, {}]]
EXPECTED value GOT EOF
not ok 3 - {"true": 1}
EXPECTED value GOT EOF
not ok 4 - Parsing package.json failed!
4 test(s) failed
FAIL: test/parse-test.sh 0
TEST: test/tokenizer-test.sh
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+"dah"
not ok 1 - "dah"
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+""
not ok 2 - ""
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1,3 @@
+[
+"dah"
+]
not ok 3 - ["dah"]
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+"   "
not ok 4 - "   "
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+" \"  "
not ok 5 - " \"  "
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1,3 @@
+[
+"dah"
+]
not ok 6 - ["dah"]
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+123
not ok 7 - 123
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+123.142
not ok 8 - 123.142
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+-123
not ok 9 - -123
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+1e23
not ok 10 - 1e23
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+0.1
not ok 11 - 0.1
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+-110
not ok 12 - -110
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+-110.10
not ok 13 - -110.10
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+-110e10
not ok 14 - -110e10
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+null
not ok 15 - null
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+true
not ok 16 - true
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1 @@
+false
not ok 17 - false
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1,7 @@
+[
+null
+,
+-110e10
+,
+"null"
+]
not ok 18 - [ null   ,  -110e10, "null" ]
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1,5 @@
+{
+"e"
+:
+false
+}
not ok 19 - {"e": false}
--- -
+++ /tmp/json_ttest_expected
@@ -0,0 +1,5 @@
+{
+"e"
+:
+"string"
+}
not ok 20 - {"e": "string"}
Tokenizing package.json failed!
21 test(s) failed
FAIL: test/tokenizer-test.sh 4
TEST: test/valid-test.sh
1..13
EXPECTED value GOT EOF
--- -
+++ valid/array.parsed
@@ -0,0 +1,5 @@
+[0]    1
+[1]    2
+[2]    3
+[3]    "hello"
+[]     [1,2,3,"hello"]
not ok 1 - valid/array.json
EXPECTED value GOT EOF
--- -
+++ valid/embedded.parsed
@@ -0,0 +1,2 @@
+["foo"]        "{\"foo\":\"bar\"}"
+[]     {"foo":"{\"foo\":\"bar\"}"}
not ok 2 - valid/embedded.json
EXPECTED value GOT EOF
--- -
+++ valid/empty_array.parsed
@@ -0,0 +1 @@
+[]     []
not ok 3 - valid/empty_array.json
EXPECTED value GOT EOF
--- -
+++ valid/empty_object.parsed
@@ -0,0 +1 @@
+[]     {}
not ok 4 - valid/empty_object.json
EXPECTED value GOT EOF
--- -
+++ valid/many_object.parsed
@@ -0,0 +1,3 @@
+["key1"]       "string"
+["key2"]       3573
+[]     {"key1":"string","key2":3573}
not ok 5 - valid/many_object.json
EXPECTED value GOT EOF
--- -
+++ valid/nested_array.parsed
@@ -0,0 +1,9 @@
+[0]    1
+[1]    []
+[2,0]  4
+[2,1]  "hello"
+[2,2]  {}
+[2]    [4,"hello",{}]
+[3,"array"]    []
+[3]    {"array":[]}
+[]     [1,[],[4,"hello",{}],{"array":[]}]
not ok 6 - valid/nested_array.json
EXPECTED value GOT EOF
--- -
+++ valid/nested_object.parsed
@@ -0,0 +1,5 @@
+["object","key"]       "value"
+["object","empty"]     {}
+["object"]     {"key":"value","empty":{}}
+["number"]     5
+[]     {"object":{"key":"value","empty":{}},"number":5}
not ok 7 - valid/nested_object.json
EXPECTED value GOT EOF
--- -
+++ valid/number.parsed
@@ -0,0 +1 @@
+[]     3
not ok 8 - valid/number.json
EXPECTED value GOT EOF
--- -
+++ valid/object.parsed
@@ -0,0 +1,2 @@
+["key"]        "Value"
+[]     {"key":"Value"}
not ok 9 - valid/object.json
EXPECTED value GOT EOF
--- -
+++ valid/string.parsed
@@ -0,0 +1 @@
+[]     "hello this is a string"
not ok 10 - valid/string.json
EXPECTED value GOT EOF
--- -
+++ valid/string_in_array.parsed
@@ -0,0 +1,2 @@
+[0]    "hello this is a string"
+[]     ["hello this is a string"]
not ok 11 - valid/string_in_array.json
EXPECTED value GOT EOF
--- -
+++ valid/string_in_object.parsed
@@ -0,0 +1,2 @@
+["key"]        "hello this is a string"
+[]     {"key":"hello this is a string"}
not ok 12 - valid/string_in_object.json
EXPECTED value GOT EOF
--- -
+++ valid/tab_escape.parsed
@@ -0,0 +1 @@
+[]     "hello\tworld"
not ok 13 - valid/tab_escape.json
13 test(s) failed
FAIL: test/valid-test.sh 25
FAILURE 1 / 4

You could debug this by adding.

By adding what? Or do you mean what you said afterwards?

cd /JSON.sh-master/
. JSON.sh
tokenize < package.json

Didn't produce any output.

[dominictarr]

right okay. It looks like you don't have a compatible egrep or awk egrep.

see this function here:
https://github.com/dominictarr/JSON.sh/blob/master/JSON.sh#L63-L91

which picks a suitable grep from what might exist....
hmm, just noticed that it goes and puts a grep inside $GREP but then doesn't use that on the thing that removes the spaces afterwards.

what happens if you change
https://github.com/dominictarr/JSON.sh/blob/master/JSON.sh#L90

to $GREP "$STRING|$NUMBER|$KEYWORD|$SPACE|." | $GREP -v "^$SPACE$"

otherwise, what grep do you have?

[dominictarr]

I ment to say "... adding logging to the various stages" but it looks like we have found a lead now, since tokenize is not outputting anything.

[philippgille]

I replaced egrep by $GREP, then:

$ . JSON.sh
$ tokenize < package.json

Still no output. In both if-statements the condition is true (checked with echo).

$ grep --version
GNU grep 2.5.3

As I'm using MobaXterm and not Cygwin directly, I thought maybe that might be an issue, but I just installed Cygwin 1.7.35 and tried out tokenize, with the same results (no output). The real Cygwin installation seems to use grep (GNU grep) 2.21. So the versions differ, but the results don't.

[dominictarr]

do you have the egrep command?

[dominictarr]

there is something inside tokenize that isn't working. if you can figure out what it is, we can fix it,
but since it's on your system, you'll need to reproduce it.