wrong Block.Header.AppHash

[kaber2]

I'm running v1.0.12 on 14 servers. All of these suddenly experience:

ERR Error in validation err="wrong Block.Header.AppHash. Expected E2B736489350C51710CE727DF19833B19A8A9F59A4DA64ED9CF2AB647F8B4FE3, got BDDF7A838C74542704B60DDA0FC8EB5803A0F5AAC0456C66B3BAF4D931DD8E2F" module=blockchain server=node

Given the recent gas price problems with metamask, is there any other known problem that could lead to this?

[mr-kenikh]

@kaber2 https://discord.com/channels/783264383978569728/907082701708853348/1184886452602024107

[kaber2]

Thanks, but that link just gives me a weird error message about "text channels". Care to post the content here?

[mr-kenikh]

@kaber2

⚠️⚠️ [IMPORTANT - CRONOS EVM CHAIN] ⚠️⚠️

Hello everone - We are rolling out a breaking upgrade cronos-v1.0.11-hardfork / cronos-v1.0.13-hardfork

🕑 Effective time:
The scheduled upgrade will be happening at block https://cronoscan.com/block/countdown/11608760
( Est 8 AM HKT, Friday 15th December )

❔ What’s this upgrade about:
This upgrade will update the feemarket parameter of Cronos mainnet, including:
Setting the minimum base fee to be 10,000 gwei
Introducing a more efficient fee market based on on-chain traffic.

🔨 Action required:
Node host must apply a new binary at the upgrade height; otherwise, it will stop operating;

Binary changes
CAN be implemented before the upgrade height, and they will automatically take effect at the upgrade height; BUT --
-❗ BUT❗ This early change will impact the response of eth_gasPrice, causing it to return an insufficient number of gas prices for transactions before the upgrade height; If you are not serving this call, feel free to already go ahead.

Expected behaviour
After the upgrade block 11608760
The minimum base fee will be 10,000 gwei
The fee market of Cronos in the EIP-1559 manner will be more effective based on on-chain traffic

Note
To auto-pause the node before the upgrade height, we can restart cronosd with the flag --halt-height 11608759.
Cronos Block Countdown
Block countdown on the Cronos Chain blockchain.
Binaries

Depending on your current version - For v1.0.12 onward, please use cronos-v1.0.13-hardfork, otherwise please use cronos-v1.0.11-hardfork.

cronos-v1.0.11-hardfork-arm64
https://cronos-binary.s3.ap-southeast-1.amazonaws.com/cronos-v1.0.11-hardfork/cronos-v1.0.11-hardfork-arm64.tar.gz

cronos-v1.0.11-hardfork-x86_64
https://cronos-binary.s3.ap-southeast-1.amazonaws.com/cronos-v1.0.11-hardfork/cronos-v1.0.11-hardfork-x86_64.tar.gz

cronos-v1.0.13-hardfork-arm64
https://cronos-binary.s3.ap-southeast-1.amazonaws.com/cronos-v1.0.13-hardfork/cronos-v1.0.13-hardfork-arm64.tar.gz

cronos-v1.0.13-hardfork-x86_64
https://cronos-binary.s3.ap-southeast-1.amazonaws.com/cronos-v1.0.13-hardfork/cronos-v1.0.13-hardfork-x86_64.tar.gz

cronos-v1.0.13-hardfork-windows
https://cronos-binary.s3.ap-southeast-1.amazonaws.com/cronos-v1.0.13-hardfork/cronos-v1.0.13-hardfork-windows.tar.gz

[kaber2]

Seriously, WTF? There is no release here, no announcement here, just on some weird gaming chat site, binaries are provided on some other random site, additionally there don't even seem to be the sources available for the binaries, in clear violation of the GPL license.

If you consider this appropriate for running your blockchain, I'll pass and deactivate my nodes.

[mr-kenikh]

Seriously, WTF? There is no release here, no announcement here, just on some weird gaming chat site, binaries are provided on some other random site, additionally there don't even seem to be the sources available for the binaries, in clear violation of the GPL license.

If you consider this appropriate for running your blockchain, I'll pass and deactivate my nodes.

And what did you expect from a centralised blockchain?
You can get source code here https://github.com/yihuang/cronos/tree/release/v1.0.x

[kaber2]

I expect at least a minimum of professionalism. This weird idea that node operators have time to hang out on 50 different colorful chat channels to get vital information is completely absurd. Github is where the code is published and this is where important information belongs.

I also expect them to not violate the copyrights of all the code they are using, and expecting node operators to run binary crap from random websites without accompanying sources.

Thanks for the link!

[kaber2]

To top it off, this is crypto, and these clowns didn't even manage to sign their binary crap, announced in a chat channel and pointing to a random website.

Way to go guys to keep your users secure.

[mr-kenikh]

@kaber2 sorry for that, and yes, this is the second such story, but last time there wasn't even source code (due to vulnerability). Btw, I am not affiliated with cronos labs

[kaber2]

Yeah, I remember.

Btw, I am not affiliated with cronos labs

I got that :)

[yihuang]

sorry for the confusion, it's an emergent upgrade, the code is in the release/v1.0.x branch, will do a release soon(#1262)

[allenwq]

hi @yihuang , I still face the same error after switched to v1.0.11-hardfork. I was using 1.0.8 prior to the fork. appreciate your advice, thanks!

11:54AM INF Replay: Vote blockID={"hash":"94A71A35C738742AB3838FA41B536744356DD$
277F2A8E4628E7D4A7F2659419","parts":{"hash":"27E749A35D2098CA3CA265A8E3B853B527$
F1DB921057B4275E0A63CFD98260B","total":1}} height=11608761 module=consensus pee$
=f46e80e846f4325c3a2ef3a6618b6aa92a382ab4 round=0 server=node type=1
panic: precommit step; +2/3 prevoted for an invalid block: wrong Block.Header.A$
pHash. Expected E2B736489350C51710CE727DF19833B19A8A9F59A4DA64ED9CF2AB647F8B4FE
3, got BDDF7A838C74542704B60DDA0FC8EB5803A0F5AAC0456C66B3BAF4D931DD8E2F

[yihuang]

hi @yihuang , I still face the same error after switched to v1.0.11-hardfork. I was using 1.0.8 prior to the fork. appreciate your advice, thanks!

11:54AM INF Replay: Vote blockID={"hash":"94A71A35C738742AB3838FA41B536744356DD$

277F2A8E4628E7D4A7F2659419","parts":{"hash":"27E749A35D2098CA3CA265A8E3B853B527$

F1DB921057B4275E0A63CFD98260B","total":1}} height=11608761 module=consensus pee$

=f46e80e846f4325c3a2ef3a6618b6aa92a382ab4 round=0 server=node type=1

panic: precommit step; +2/3 prevoted for an invalid block: wrong Block.Header.A$

pHash. Expected E2B736489350C51710CE727DF19833B19A8A9F59A4DA64ED9CF2AB647F8B4FE

3, got BDDF7A838C74542704B60DDA0FC8EB5803A0F5AAC0456C66B3BAF4D931DD8E2F

You'll need to do a cronosd rollback to recover, because the wrong state is already committed, which'll take some time if you are using iavl with fast node index enabled.

[yihuang]

v1.0.14 is released

[kaber2]

So what was the emergency? Given that a fee increase has only one possible goal, reduction in transaction rate, how can that be an emergency? Any well designed system would keep operating at the transaction rate it actually can handle.

[allenwq]

@yihuang thanks for the help! Anyway to speed up the rollback? My cronosd rollback is running for more than 12 hours and it's still not finished 😭

[yihuang]

@yihuang thanks for the help! Anyway to speed up the rollback? My cronosd rollback is running for more than 12 hours and it's still not finished 😭

Disable iavl fast node index can make rollback a lot faster, the rollback must recreate that index , but without the index, the state machine runs slower.

[allenwq]

@yihuang thanks for the help! Anyway to speed up the rollback? My cronosd rollback is running for more than 12 hours and it's still not finished 😭

Disable iavl fast node index can make rollback a lot faster, the rollback must recreate that index , but without the index, the state machine runs slower.

Understand, any shell command to disable and enable the iavl index?

[yihuang]

@yihuang thanks for the help! Anyway to speed up the rollback? My cronosd rollback is running for more than 12 hours and it's still not finished 😭

Disable iavl fast node index can make rollback a lot faster, the rollback must recreate that index , but without the index, the state machine runs slower.

Understand, any shell command to disable and enable the iavl index?

It's in the app.toml, iavl-disable-fastnode-index