Change JSON library to improve performance

[hjweddie]

我们在生产环境解析full image binlog时，cpu使用率非常高。进行pprof之后发现在json.Unmarshal函数的消费非常大。
我们是否可以使用性能更优的json库来替换encoding/json呢？

We found heavy cpu usage when parsing full image binlog. As our pprof result, find that encoding/json.Unmarshal consumes much cpu time. Is it posible to use jsonitor or other json libary for better json unmarshal performance ?

[hjweddie]

This is one of the pprof result

[lance6716]

lgtm.

And we have introduced a new API in #737 . User can pass a function that checking table is not needed to be replicated in RowsEventDecodeFunc then skip decoding the data of that table, like #831

[lance6716]

I'll check the quality of github.com/json-iterator some times later

[lance6716]

Can you try RowsEventDecodeFunc first? My main concern about github.com/json-iterator/go is it's no longer maintained and some bug-fixing PRs are not merged

[lance6716]

Can you try RowsEventDecodeFunc first? My main concern about github.com/json-iterator/go is it's no longer maintained and some bug-fixing PRs are not merged

@hjweddie Hi do you have any response for it?

[hjweddie]

Can you try RowsEventDecodeFunc first? My main concern about github.com/json-iterator/go is it's no longer maintained and some bug-fixing PRs are not merged

@hjweddie Hi do you have any response for it?

sorry for late, I will try it in this week

[hjweddie]

Can you try RowsEventDecodeFunc first? My main concern about github.com/json-iterator/go is it's no longer maintained and some bug-fixing PRs are not merged

I agree with your concern, maybe we could use another more active json repo, like https://github.com/goccy/go-json

[lance6716]

You can inspect the workload to see which table the JSON parsing comes from. If the table is not needed, we can use RowsEventDecodeFunc like #831 to skip decode it. But if the JSON parsing is a must, we have to switch to a better JSON library.

[hjweddie]

In fact we use jsonitor in production for years and did not encounter Marshal/Unmarshal failures. Maybe it is with good functionality

[hjweddie]

You can inspect the workload to see which table the JSON parsing comes from. If the table is not needed, we can use RowsEventDecodeFunc like #831 to skip decode it. But if the JSON parsing is a must, we have to switch to a better JSON library.

I agree

[hjweddie]

You can inspect the workload to see which table the JSON parsing comes from. If the table is not needed, we can use RowsEventDecodeFunc like #831 to skip decode it. But if the JSON parsing is a must, we have to switch to a better JSON library.

I have reviewed my production program, it is a monitor towards binlog size to get original data difference, is it necessary to parse it ? I guest it may be necessary ?

[lance6716]

@atercattus PTAL

[atercattus]

We also have some incompatibilities with stdlib in github.com/json-iterator/go. So I cannot recommend it.

@hjweddie, can you show your pprof with github.com/goccy/go-json? Is performance difference significant?

[hjweddie]

I change to use github.com/goccy/go-json in our production environment, it seemds that json unmarshal do not consume cpu heavily

[hjweddie]

I have replaced github.com/json-iterator/go with github.com/goccy/go-json in this pr

[lance6716]

ping @atercattus

[atercattus]

Sorry for the late response. pprof with go-json looks good. So good that I don't see this library at all :)

[lance6716]

I think I can generate a binlog file that's full of JSON change, and let this library consume it, to see the performance improvement. I'm busy now, will do it in next week

[hjweddie]

After our production process run for a few days, here is the result from go tool pprof /debug/pprof/profile?seconds=50

It seemds that json unmarshal do not consume cpu heavily

[lance6716]

I started a MySQL in localhost, and generate some JSON INSERT to get a 1G binlog file. Then I compare the time of reading and parsing that file

master:
1m42.265181011s

this PR:
1m18.27619239s

so LGTM

[slotix]

I confirm that goccy/go-json is working fine. Please disregard this message.

Following your concern regarding exchanging standard encoding/json with goccy/go-json, I've switched to goccy in my repository and noticed some peculiar behavior during the marshaling/unmarshaling of data.

I'm encountering unusual errors such as:

When encoding :

encoder: opcode  has not been implemented

It points somewhere to

https://github.com/goccy/go-json/blob/df897aec9dc4228e585e8127b4db026d506d2b3c/internal/encoder/vm/util.go#L46

When decoding :

invalid character 'Ü' looking for the beginning of a value
invalid character '«' looking for the beginning of a value

<error(*github.com/goccy/go-json/internal/errors.SyntaxError)>)
<*github.com/goccy/go-json/internal/errors.SyntaxError>(0xc000a9ae58)
msg: "invalid character '×' looking for beginning of value"
Offset: 28724

Upon reverting to standard encoding/json, I no longer experience similar errors. It's worth noting that there are no issues with github.com/json-iterator/go. Please be aware that I am not affiliated with any of the JSON libraries; I'm just using go-mysql in my project and am aware of potential problems with goccy.

[lance6716]

Thanks for reporting @slotix , can you find the JSON content when error happens? We can check if the error pattern will also occur in MySQL binlog.

We also have some incompatibilities with stdlib in github.com/json-iterator/go. So I cannot recommend it.

@hjweddie, can you show your pprof with github.com/goccy/go-json? Is performance difference significant?

It's worth noting that there are no issues with github.com/json-iterator/go.

@atercattus not sure if we should give github.com/goccy/go-json a chance 😂

[atercattus]

Bad news...

@slotix, could you catch this json? Maybe I can fix goccy or at least I’ll understand what’s wrong.

Upd. I found goccy/go-json#493 :)

[slotix]

Please wait for a moment. I need to conduct more thorough testing. Interestingly, for some reason, I haven't been able to reproduce this bug at all since this morning :)

P.S. Please disregard my initial message about errors caused by goccy/go-json.

I thoroughly tested my code throughout the day, marshaling/unmarshaling over 150GB of randomly generated JSON data, and encountered none of the errors reported yesterday.

I suspect that some ghost instances of my services may have caused these strange problems.