Problem: no binary genesis import & export

[leejw51crypto]

currently, genesis import & export is done plain json text file.
which is large & inefficient to handle.

1. add import & export with binary format
2. sqlite would be good fit , because user can also check the contents

[JayT106]

currently, genesis import & export is done plain json text file. which is large & inefficient to handle.

@leejw51crypto
Could you explain in more detail the inefficiency? do you mean the import/export speed?

[JayT106]

the exporting state at height 3104880:

exporting... module genesis genutil
exported module genesis genutil, size 14, time: 29.858µs
exporting... module genesis auth
exported module genesis auth, size 255533801, time: 18m19.766358778s
exporting... module genesis vesting
exported module genesis vesting, size 2, time: 2.002µs
exporting... module genesis bank
exported module genesis bank, size 63231917, time: 9m30.362255526s
exporting... module genesis capability
exported module genesis capability, size 1118, time: 6.782134ms
exporting... module genesis crisis
exported module genesis crisis, size 50, time: 182.056µs
exporting... module genesis gov
exported module genesis gov, size 2832, time: 10.035674ms
exporting... module genesis mint
exported module genesis mint, size 312, time: 241.683µs
exporting... module genesis slashing
exported module genesis slashing, size 7081945, time: 4m4.39138446s
exporting... module genesis distribution
exported module genesis distribution, size 33976, time: 1.175575771s
exporting... module genesis staking
exported module genesis staking, size 33249, time: 475.912321ms
exporting... module genesis upgrade
exported module genesis upgrade, size 2, time: 1.723µs
exporting... module genesis evidence
exported module genesis evidence, size 15, time: 43.585µs
exporting... module genesis ibc
exported module genesis ibc, size 297076853, time: 25m11.473207169s
exporting... module genesis params
exported module genesis params, size 0, time: 1.256µs
exporting... module genesis feegrant
exported module genesis feegrant, size 6978, time: 56.168646ms
exporting... module genesis authz
exported module genesis authz, size 876, time: 229.711µs
exporting... module genesis transfer
exported module genesis transfer, size 3140, time: 5.443416ms
exporting... module genesis evm
exported module genesis evm, size 4967554826, time: 17h16m30.357516549s
exporting... module genesis feemarket
exported module genesis feemarket, size 202, time: 36.02913ms
exporting... module genesis cronos
exported module genesis cronos, size 6018, time: 424.298829ms
appExporter time 18h27m23.268923683s

starting marshal genesis
done marshal genesis, time 2m32.038029189s, size: 5590573903
starting marshal sort genesis
done marshal sort genesis, time 5m43.869593346s, size: 5590573903
exporting Binary
exported Binary, size: 5590573903

[JayT106]

The long-term solution might be using the ORM feature after Cosmos SDK v0.46.x. However, it will need the whole module's state migration using the ORM table and the known downgrading DB performance issue.

Currently trying to export the genesis state to files in each module and save it to a genesis/[module] folder. Also, splits the state into several files if the state is huge, like evm, ibc, auth. Therefore, it can save the memory requirement when doing the import/export. The current genesis export consumes more than 64GB RAM to be able to finish. WIP

[JayT106]

testing the genesis export to genesis/[module], it shows we can reduce some space and time usage with the same dataset.

Time: 15h56m v.s. 18h27m
Space: 4.6G v.s. 5.6G

For the OOM issue, it relates to tendermint/tm-db#272 , we will need to limit the file open size to around 1000

[JayT106]

submit PR in the SDK, but it also requires the other project to implement the methods.
cosmos/cosmos-sdk#13032