Auto-schema on INSERT: incorrect JSON response for big IDs

[kakserpom]

Bug Description:

Reproduction:

 curl -sX POST http://localhost:9308/insert -d'{"index":"foo","id":18446744073709551615,"doc":{"long":10.0,"lat":10.0,"description":"123","short_description":"few words..."}}'

{"_index":"foo","_id":18446744073709551616.000000,"created":true,"result":"created","status":201}%

_id becomes float for some reason.
It happens only if the target index doesn't exist and if id is big enough.

Manticore Search Version:

6.3.0

Operating System Version:

Docker

Have you tried the latest development version?

None

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• Changelog updated

[donhardman]

The implementation of the core library for handling BigInt with overflow support for signed variants is now available at: manticoresoftware/buddy-core#64

We should proceed to update the auto schema plugin in Buddy using this library. I believe @Nick-S-2018 has more context and can assist with the upgrade.

The Struct object is designed to be used as follows:

// Create a Struct from JSON
$struct = Struct::fromJson(...);

// Access data as an array (BigInts will be strings)
$struct->toArray();
// or
$struct['key'];

// Convert back to JSON while preserving BigInts
$struct->toJson();

// Replace a BigInt value
$struct['id'] = '...';

// Create a new field and set its type to BigInt
$struct['nested']['new_field'] = '1111';
$struct->addBigIntField('nested.new_field');

This implementation allows for easy handling of BigInt values, ensuring they're properly encoded in JSON output and accessible as strings when needed. 😊

[Nick-S-2018]

As discussed with@donhardman, it's turned out that the proposed implementation will require changes in buddy-core too, so we need to decide on the most efficient way to do that first.

[sanikolaev]

@Nick-S-2018 pls let @donhardman what's required from him within this issue.

[Nick-S-2018]

Probably, we need to decode input and encode output not in the plugins, as we tried before, but by using some core's Request/Response methods for that purpose.

[donhardman]

@Nick-S-2018 please review the pull request in core: manticoresoftware/buddy-core#75

I have also added modifications to the main pull request. We should make sure that everything works fine with the new core. Also, please do some tests to ensure inserts work as expected.

[Nick-S-2018]

Now Buddy passes tests and handles bigints correctly. However, after Buddy response is passed back to daemon, they're still got converted to incorrect values. E.g.:

curl -sX POST http://localhost:9308/cli -d 'create table test(title text)'
curl -sX POST http://localhost:9308/insert -d '{"index":"test","id":18446744073709551615,"doc":{"title":"a"}}'

{"table":"test","_id":18446744073709551615,"created":true,"result":"created","status":201}

// Performing the same request through Buddy's auto-schema
curl -sX POST http://localhost:9308/cli -d 'drop table test'
curl -sX POST http://localhost:9308/insert -d '{"index":"test","id":18446744073709551615,"doc":{"title":"a"}}'

{"table":"test","_id":9223372036854775807,"created":true,"result":"created","status":201}

//Response from Buddy to daemon:
[BUDDY] [671b63c0d8e873.36716686] response data: {"version":3,"type":"json response","message":{"table":"test","_id":18446744073709551615,"created":true,"result":"created","status":201},"meta":null,"error_code":200}

The example can be reproduced with:

• Manticore 6.3.7 bb1ca3193@24102419 dev
• Buddy branch: https://github.com/manticoresoftware/manticoresearch-buddy/tree/feature/json_bigint_support
• Buddy-core branch: https://github.com/manticoresoftware/buddy-core/tree/feature/json_bigint_support

[tomatolog]

I fixed the issue at daemon f01a478 by switching JSON parser from BJson to CJson at buddy communication. Now parsing of the buddy replies work the same way as parser of the HTTP JSON requests. That fixes the issue at daemon with large integer.

However buddy main branch and buddy-core branch should be merged.

[Nick-S-2018]

It looks that response format from daemon to Buddy is changed to JSON string now which leads to Buddy tests failures. E.g:

curl localhost:9408/cli -d 'show queries'
"+-----+--------------+----------+----------+------------------+\n| id  | query        | time     | protocol | host             |\n+-----+--------------+----------+----------+------------------+\n| 181 | show queries | 32ms ago | http     | 172.17.0.1:59686 |\n| 182 | select       | 1ms ago  | http     | 127.0.0.1:42182  |\n| 4   | show_tables  | 9us      | http     | 127.0.0.1:33860  |\n+-----+--------------+----------+----------+------------------+\n3 rows in set (0.005 sec)\n\n"

[tomatolog]

I checked reply from the daemon to buddy for buddy request

SELECT * FROM @@system.sessions"

but see no difference

here is a dump of the request prior to fix

48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d  HTTP/1.1 200 OK.
0a 53 65 72 76 65 72 3a 20 36 2e 33 2e 37 20 36  .Server: 6.3.7 6
33 39 32 62 64 33 31 64 40 32 34 31 30 31 38 31  392bd31d@2410181
31 20 64 65 76 20 28 63 6f 6c 75 6d 6e 61 72 20  1 dev (columnar 
32 2e 33 2e 31 20 62 61 39 65 32 38 33 40 32 34  2.3.1 ba9e283@24
31 30 30 39 31 35 29 20 28 73 65 63 6f 6e 64 61  100915) (seconda
72 79 20 32 2e 33 2e 31 20 62 61 39 65 32 38 33  ry 2.3.1 ba9e283
40 32 34 31 30 30 39 31 35 29 20 28 6b 6e 6e 20  @24100915) (knn 
32 2e 33 2e 31 20 62 61 39 65 32 38 33 40 32 34  2.3.1 ba9e283@24
31 30 30 39 31 35 29 20 28 62 75 64 64 79 20 76  100915) (buddy v
33 2e 30 2e 31 29 0d 0a 43 6f 6e 74 65 6e 74 2d  3.0.1)..Content-
54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 69 6f  Type: applicatio
6e 2f 6a 73 6f 6e 3b 20 63 68 61 72 73 65 74 3d  n/json; charset=
55 54 46 2d 38 0d 0a 43 6f 6e 74 65 6e 74 2d 4c  UTF-8..Content-L
65 6e 67 74 68 3a 20 35 38 35 0d 0a 0d 0a 5b 7b  ength: 585....[{
0a 22 63 6f 6c 75 6d 6e 73 22 3a 5b 7b 22 69 64  ."columns":[{"id
22 3a 7b 22 74 79 70 65 22 3a 22 6c 6f 6e 67 20  ":{"type":"long 
6c 6f 6e 67 22 7d 7d 2c 7b 22 70 72 6f 74 6f 22  long"}},{"proto"
3a 7b 22 74 79 70 65 22 3a 22 73 74 72 69 6e 67  :{"type":"string
22 7d 7d 2c 7b 22 73 74 61 74 65 22 3a 7b 22 74  "}},{"state":{"t
79 70 65 22 3a 22 73 74 72 69 6e 67 22 7d 7d 2c  ype":"string"}},
7b 22 68 6f 73 74 22 3a 7b 22 74 79 70 65 22 3a  {"host":{"type":
22 73 74 72 69 6e 67 22 7d 7d 2c 7b 22 63 6f 6e  "string"}},{"con
6e 69 64 22 3a 7b 22 74 79 70 65 22 3a 22 6c 6f  nid":{"type":"lo
6e 67 20 6c 6f 6e 67 22 7d 7d 2c 7b 22 6b 69 6c  ng long"}},{"kil
6c 65 64 22 3a 7b 22 74 79 70 65 22 3a 22 73 74  led":{"type":"st
72 69 6e 67 22 7d 7d 2c 7b 22 6c 61 73 74 20 63  ring"}},{"last c
6d 64 20 74 69 6d 65 22 3a 7b 22 74 79 70 65 22  md time":{"type"
3a 22 73 74 72 69 6e 67 22 7d 7d 2c 7b 22 6c 61  :"string"}},{"la
73 74 20 63 6d 64 22 3a 7b 22 74 79 70 65 22 3a  st cmd":{"type":
22 73 74 72 69 6e 67 22 7d 7d 5d 2c 0a 22 64 61  "string"}}],."da
74 61 22 3a 5b 0a 7b 22 69 64 22 3a 32 2c 22 70  ta":[.{"id":2,"p
72 6f 74 6f 22 3a 22 6d 79 73 71 6c 22 2c 22 73  roto":"mysql","s
74 61 74 65 22 3a 22 71 75 65 72 79 22 2c 22 68  tate":"query","h
6f 73 74 22 3a 22 31 32 37 2e 30 2e 30 2e 31 3a  ost":"127.0.0.1:
36 32 34 33 32 22 2c 22 63 6f 6e 6e 69 64 22 3a  62432","connid":
35 32 30 2c 22 6b 69 6c 6c 65 64 22 3a 22 30 22  520,"killed":"0"
2c 22 6c 61 73 74 20 63 6d 64 20 74 69 6d 65 22  ,"last cmd time"
3a 22 36 75 73 22 2c 22 6c 61 73 74 20 63 6d 64  :"6us","last cmd
22 3a 22 73 68 6f 77 20 71 75 65 72 69 65 73 22  ":"show queries"
7d 2c 0a 7b 22 69 64 22 3a 31 2c 22 70 72 6f 74  },.{"id":1,"prot
6f 22 3a 22 68 74 74 70 22 2c 22 73 74 61 74 65  o":"http","state
22 3a 22 71 75 65 72 79 22 2c 22 68 6f 73 74 22  ":"query","host"
3a 22 31 32 37 2e 30 2e 30 2e 31 3a 36 32 34 32  :"127.0.0.1:6242
37 22 2c 22 63 6f 6e 6e 69 64 22 3a 35 31 39 2c  7","connid":519,
22 6b 69 6c 6c 65 64 22 3a 22 30 22 2c 22 6c 61  "killed":"0","la
73 74 20 63 6d 64 20 74 69 6d 65 22 3a 22 32 36  st cmd time":"26
35 75 73 20 61 67 6f 22 2c 22 6c 61 73 74 20 63  5us ago","last c
6d 64 22 3a 22 73 65 6c 65 63 74 22 7d 0a 5d 2c  md":"select"}.],
0a 22 74 6f 74 61 6c 22 3a 32 2c 0a 22 65 72 72  ."total":2,."err
6f 72 22 3a 22 22 2c 0a 22 77 61 72 6e 69 6e 67  or":"",."warning
22 3a 22 22 0a 7d 5d cd cd cd cd cd cd cd cd cd  ":"".}]

here is a dump of the request at the master version with the fix

48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d  HTTP/1.1 200 OK.
0a 53 65 72 76 65 72 3a 20 36 2e 33 2e 37 20 36  .Server: 6.3.7 6
33 39 32 62 64 33 31 64 40 32 34 31 30 31 38 31  392bd31d@2410181
31 20 64 65 76 20 28 63 6f 6c 75 6d 6e 61 72 20  1 dev (columnar 
32 2e 33 2e 31 20 62 61 39 65 32 38 33 40 32 34  2.3.1 ba9e283@24
31 30 30 39 31 35 29 20 28 73 65 63 6f 6e 64 61  100915) (seconda
72 79 20 32 2e 33 2e 31 20 62 61 39 65 32 38 33  ry 2.3.1 ba9e283
40 32 34 31 30 30 39 31 35 29 20 28 6b 6e 6e 20  @24100915) (knn 
32 2e 33 2e 31 20 62 61 39 65 32 38 33 40 32 34  2.3.1 ba9e283@24
31 30 30 39 31 35 29 20 28 62 75 64 64 79 20 76  100915) (buddy v
33 2e 30 2e 31 29 0d 0a 43 6f 6e 74 65 6e 74 2d  3.0.1)..Content-
54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 69 6f  Type: applicatio
6e 2f 6a 73 6f 6e 3b 20 63 68 61 72 73 65 74 3d  n/json; charset=
55 54 46 2d 38 0d 0a 43 6f 6e 74 65 6e 74 2d 4c  UTF-8..Content-L
65 6e 67 74 68 3a 20 37 32 36 0d 0a 0d 0a 5b 7b  ength: 726....[{
0a 22 63 6f 6c 75 6d 6e 73 22 3a 5b 7b 22 69 64  ."columns":[{"id
22 3a 7b 22 74 79 70 65 22 3a 22 6c 6f 6e 67 20  ":{"type":"long 
6c 6f 6e 67 22 7d 7d 2c 7b 22 70 72 6f 74 6f 22  long"}},{"proto"
3a 7b 22 74 79 70 65 22 3a 22 73 74 72 69 6e 67  :{"type":"string
22 7d 7d 2c 7b 22 73 74 61 74 65 22 3a 7b 22 74  "}},{"state":{"t
79 70 65 22 3a 22 73 74 72 69 6e 67 22 7d 7d 2c  ype":"string"}},
7b 22 68 6f 73 74 22 3a 7b 22 74 79 70 65 22 3a  {"host":{"type":
22 73 74 72 69 6e 67 22 7d 7d 2c 7b 22 63 6f 6e  "string"}},{"con
6e 69 64 22 3a 7b 22 74 79 70 65 22 3a 22 6c 6f  nid":{"type":"lo
6e 67 20 6c 6f 6e 67 22 7d 7d 2c 7b 22 6b 69 6c  ng long"}},{"kil
6c 65 64 22 3a 7b 22 74 79 70 65 22 3a 22 73 74  led":{"type":"st
72 69 6e 67 22 7d 7d 2c 7b 22 6c 61 73 74 20 63  ring"}},{"last c
6d 64 20 74 69 6d 65 22 3a 7b 22 74 79 70 65 22  md time":{"type"
3a 22 73 74 72 69 6e 67 22 7d 7d 2c 7b 22 6c 61  :"string"}},{"la
73 74 20 63 6d 64 22 3a 7b 22 74 79 70 65 22 3a  st cmd":{"type":
22 73 74 72 69 6e 67 22 7d 7d 5d 2c 0a 22 64 61  "string"}}],."da
74 61 22 3a 5b 0a 7b 22 69 64 22 3a 32 2c 22 70  ta":[.{"id":2,"p
72 6f 74 6f 22 3a 22 6d 79 73 71 6c 22 2c 22 73  roto":"mysql","s
74 61 74 65 22 3a 22 71 75 65 72 79 22 2c 22 68  tate":"query","h
6f 73 74 22 3a 22 31 32 37 2e 30 2e 30 2e 31 3a  ost":"127.0.0.1:
36 32 36 30 31 22 2c 22 63 6f 6e 6e 69 64 22 3a  62601","connid":
32 36 2c 22 6b 69 6c 6c 65 64 22 3a 22 30 22 2c  26,"killed":"0",
22 6c 61 73 74 20 63 6d 64 20 74 69 6d 65 22 3a  "last cmd time":
22 36 75 73 22 2c 22 6c 61 73 74 20 63 6d 64 22  "6us","last cmd"
3a 22 73 68 6f 77 20 71 75 65 72 69 65 73 22 7d  :"show queries"}
2c 0a 7b 22 69 64 22 3a 33 2c 22 70 72 6f 74 6f  ,.{"id":3,"proto
22 3a 22 68 74 74 70 22 2c 22 73 74 61 74 65 22  ":"http","state"
3a 22 71 75 65 72 79 22 2c 22 68 6f 73 74 22 3a  :"query","host":
22 31 32 37 2e 30 2e 30 2e 31 3a 36 32 36 30 32  "127.0.0.1:62602
22 2c 22 63 6f 6e 6e 69 64 22 3a 32 37 2c 22 6b  ","connid":27,"k
69 6c 6c 65 64 22 3a 22 30 22 2c 22 6c 61 73 74  illed":"0","last
20 63 6d 64 20 74 69 6d 65 22 3a 22 33 39 30 75   cmd time":"390u
73 20 61 67 6f 22 2c 22 6c 61 73 74 20 63 6d 64  s ago","last cmd
22 3a 22 73 65 6c 65 63 74 22 7d 2c 0a 7b 22 69  ":"select"},.{"i
64 22 3a 31 2c 22 70 72 6f 74 6f 22 3a 22 68 74  d":1,"proto":"ht
74 70 22 2c 22 73 74 61 74 65 22 3a 22 6e 65 74  tp","state":"net
5f 69 64 6c 65 22 2c 22 68 6f 73 74 22 3a 22 31  _idle","host":"1
32 37 2e 30 2e 30 2e 31 3a 36 32 35 37 33 22 2c  27.0.0.1:62573",
22 63 6f 6e 6e 69 64 22 3a 34 2c 22 6b 69 6c 6c  "connid":4,"kill
65 64 22 3a 22 30 22 2c 22 6c 61 73 74 20 63 6d  ed":"0","last cm
64 20 74 69 6d 65 22 3a 22 38 36 75 73 22 2c 22  d time":"86us","
6c 61 73 74 20 63 6d 64 22 3a 22 73 68 6f 77 5f  last cmd":"show_
73 65 74 74 69 6e 67 73 22 7d 0a 5d 2c 0a 22 74  settings"}.],."t
6f 74 61 6c 22 3a 33 2c 0a 22 65 72 72 6f 72 22  otal":3,."error"
3a 22 22 2c 0a 22 77 61 72 6e 69 6e 67 22 3a 22  :"",."warning":"
22 0a 7d 5d cd cd cd cd cd cd cd cd cd cd cd cd  ".}]

after the header body separator 0d 0a 0d 0a there are only 0a chars the same as prior the fix.

Unclear what is the issue.

[Nick-S-2018]

I was wrong with my initial assumption. It has to do with response format from daemon to the client, not from daemon to Buddy. E.g., the latest version produces:

curl -X POST localhost:9308/cli -d 'create table test(f text)'
"Query OK, 0 rows affected (0.030 sec)\n\n"

The previous version:

curl -X POST localhost:9308/cli -d 'create table test(f text)'
Query OK, 0 rows affected (0.007 sec)

Request and response from Buddy to daemon stay the same in both cases:

request data: {"type":"unknown json request","error":{"message":"","body":null},"version":3,"message":{"path_query":"/cli","body":"create table test(f text)","http_method":"POST"}}

response data: {"version":3,"type":"json response","message":"Query OK, 0 rows affected (0.030 sec)\n\n","meta":null,"error_code":200}

[tomatolog]

fixed wrong escaping of the /cli endpoint at edcdd5e

[Nick-S-2018]

Added necessary updates to Buddy in manticoresoftware/manticoresearch-buddy#335
Needs a review now. @djklim87 please, review it.

[Nick-S-2018]

Done in manticoresoftware/manticoresearch-buddy#393