firehose-antelope really slow when replaying kylin and telos blocks

[matthewdarwin]

Just start from a snapshot from a few days ago. Watch the amount of CPU use by firehose and how slow the blockchain syncs forward (3 minutes for every 10 seconds in blocks). Without firehose there, the chain runs like 10x faster (30 minutes in 10 seconds). Any optimizations to be made?

Note: this is reader+merger running together.

[matthewdarwin]

Turning off merger doesn't make it faster.

[fschoell]

Is this limited to Kylin and Telos or does this impact all chains?

[matthewdarwin]

I didn't check the others, probably applies to all?

[fschoell]

Spend some time debugging this. It seems like:

• The new code uses approx 2x the CPU time in comparison to the old code
• The increased CPU time seems to be spend on garbage collection
• The heap usage of the old code is about 2.5x the size of the new code

It seems like the old code is utilizing more heap in favor of less CPU time. I'm not sure how though.

It might also be a possibility to tweak the garbage collector here using GOGC. As it seems like:

doubling GOGC will double heap memory overheads and roughly halve GC CPU cost

[fschoell]

Some quick testing on the new code. CPU profiling for a 30s interval reports:

• around 14s of CPU time used on GOGC=100 (default)
• around 3.5s of CPU time used on GOGC=500
• around 3s of CPU time used on GOGC=1000

in comparison it's around 7-8s of CPU time on the old code without GOGC tweaking.

So obviously need to do some testing when replaying blocks instead of getting this data in live mode, but we should definitely try this. Will increase memory usage, but this shouldn't be a big issue here I assume.