fix: check pubkey or validator index known to a state

[twoeths]

Motivation

Got this error in mekong devnet found by @nflaig

Dec-05 20:57:46.624[chain]           debug: Block error slot=207552, code=BLOCK_ERROR_PRESTATE_MISSING, error=Cannot read properties of undefined (reading 'exitEpoch')

TypeError: Cannot read properties of undefined (reading 'exitEpoch')
    at CustomContainerTreeViewDU.get [as exitEpoch] (/home/nico/projects/ethereum/lodestar/node_modules/@chainsafe/ssz/src/viewDU/containerNodeStruct.ts:78:61)
    at processPendingDeposits (file:///home/nico/projects/ethereum/lodestar/packages/state-transition/src/epoch/processPendingDeposits.ts:55:37)

Description

Consider this scenario when we have multiple epoch transitions per epoch, given n is the epoch transition slot

pending deposits of the whole epoch are processed by pulling up (n-1) state, then we add pubkeys to indes2pubkey
then at slot n, we process epoch transition again, found validator index from index2pubkey but validator is not added to registry of the state, then the issue happen

To resolve it, I added isPubkeyKnown() function and isValidatorKnown utils and use them at concerned place.

Steps to test or reproduce

resync mekong?

[codecov]

Codecov Report

Attention: Patch coverage is 25.00000% with 12 lines in your changes missing coverage. Please review.

Project coverage is 48.90%. Comparing base (d55bb2d) to head (e91c9ec).
Report is 3 commits behind head on unstable.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #7284      +/-   ##
============================================
- Coverage     48.91%   48.90%   -0.02%     
============================================
  Files           601      601              
  Lines         40182    40194      +12     
  Branches       2061     2060       -1     
============================================
+ Hits          19655    19656       +1     
- Misses        20489    20500      +11     
  Partials         38       38              

[github-actions]

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: acf20f3	Previous: b5fb76c	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	1.6837 ms/op	1.9312 ms/op	0.87
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	61.685 us/op	50.117 us/op	1.23
BLS verify - blst	766.88 us/op	983.24 us/op	0.78
BLS verifyMultipleSignatures 3 - blst	1.3907 ms/op	1.6833 ms/op	0.83
BLS verifyMultipleSignatures 8 - blst	1.6464 ms/op	1.9954 ms/op	0.83
BLS verifyMultipleSignatures 32 - blst	4.8759 ms/op	6.4291 ms/op	0.76
BLS verifyMultipleSignatures 64 - blst	8.8872 ms/op	11.213 ms/op	0.79
BLS verifyMultipleSignatures 128 - blst	17.434 ms/op	17.901 ms/op	0.97
BLS deserializing 10000 signatures	674.93 ms/op	713.02 ms/op	0.95
BLS deserializing 100000 signatures	6.7070 s/op	7.1966 s/op	0.93
BLS verifyMultipleSignatures - same message - 3 - blst	891.37 us/op	1.0445 ms/op	0.85
BLS verifyMultipleSignatures - same message - 8 - blst	1.0033 ms/op	1.1363 ms/op	0.88
BLS verifyMultipleSignatures - same message - 32 - blst	1.6745 ms/op	1.7984 ms/op	0.93
BLS verifyMultipleSignatures - same message - 64 - blst	2.5710 ms/op	2.8785 ms/op	0.89
BLS verifyMultipleSignatures - same message - 128 - blst	4.3912 ms/op	4.5639 ms/op	0.96
BLS aggregatePubkeys 32 - blst	20.170 us/op	21.660 us/op	0.93
BLS aggregatePubkeys 128 - blst	70.671 us/op	74.848 us/op	0.94
notSeenSlots=1 numMissedVotes=1 numBadVotes=10	54.577 ms/op	108.24 ms/op	0.50
notSeenSlots=1 numMissedVotes=0 numBadVotes=4	51.556 ms/op	62.701 ms/op	0.82
notSeenSlots=2 numMissedVotes=1 numBadVotes=10	40.290 ms/op	52.538 ms/op	0.77
getSlashingsAndExits - default max	134.77 us/op	150.87 us/op	0.89
getSlashingsAndExits - 2k	443.65 us/op	563.77 us/op	0.79
proposeBlockBody type=full, size=empty	6.1853 ms/op	6.9553 ms/op	0.89
isKnown best case - 1 super set check	279.00 ns/op	414.00 ns/op	0.67
isKnown normal case - 2 super set checks	291.00 ns/op	451.00 ns/op	0.65
isKnown worse case - 16 super set checks	281.00 ns/op	455.00 ns/op	0.62
InMemoryCheckpointStateCache - add get delete	2.4990 us/op	3.3550 us/op	0.74
validate api signedAggregateAndProof - struct	1.3612 ms/op	1.6190 ms/op	0.84
validate gossip signedAggregateAndProof - struct	1.4344 ms/op	1.6761 ms/op	0.86
batch validate gossip attestation - vc 640000 - chunk 32	121.57 us/op	156.72 us/op	0.78
batch validate gossip attestation - vc 640000 - chunk 64	109.14 us/op	146.94 us/op	0.74
batch validate gossip attestation - vc 640000 - chunk 128	103.97 us/op	124.36 us/op	0.84
batch validate gossip attestation - vc 640000 - chunk 256	97.544 us/op	124.23 us/op	0.79
pickEth1Vote - no votes	1.0009 ms/op	1.1410 ms/op	0.88
pickEth1Vote - max votes	5.9736 ms/op	6.1047 ms/op	0.98
pickEth1Vote - Eth1Data hashTreeRoot value x2048	15.004 ms/op	13.278 ms/op	1.13
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	18.119 ms/op	26.334 ms/op	0.69
pickEth1Vote - Eth1Data fastSerialize value x2048	436.79 us/op	594.94 us/op	0.73
pickEth1Vote - Eth1Data fastSerialize tree x2048	3.0537 ms/op	4.7901 ms/op	0.64
bytes32 toHexString	376.00 ns/op	601.00 ns/op	0.63
bytes32 Buffer.toString(hex)	217.00 ns/op	243.00 ns/op	0.89
bytes32 Buffer.toString(hex) from Uint8Array	328.00 ns/op	453.00 ns/op	0.72
bytes32 Buffer.toString(hex) + 0x	219.00 ns/op	252.00 ns/op	0.87
Object access 1 prop	0.12700 ns/op	0.18200 ns/op	0.70
Map access 1 prop	0.12500 ns/op	0.14600 ns/op	0.86
Object get x1000	5.6620 ns/op	6.1910 ns/op	0.91
Map get x1000	6.1130 ns/op	6.8790 ns/op	0.89
Object set x1000	30.812 ns/op	51.666 ns/op	0.60
Map set x1000	20.804 ns/op	37.165 ns/op	0.56
Return object 10000 times	0.27210 ns/op	0.31480 ns/op	0.86
Throw Error 10000 times	3.1602 us/op	3.6703 us/op	0.86
toHex	136.97 ns/op	167.38 ns/op	0.82
Buffer.from	117.28 ns/op	156.05 ns/op	0.75
shared Buffer	76.344 ns/op	101.40 ns/op	0.75
fastMsgIdFn sha256 / 200 bytes	2.1240 us/op	2.5160 us/op	0.84
fastMsgIdFn h32 xxhash / 200 bytes	242.00 ns/op	315.00 ns/op	0.77
fastMsgIdFn h64 xxhash / 200 bytes	267.00 ns/op	315.00 ns/op	0.85
fastMsgIdFn sha256 / 1000 bytes	6.9970 us/op	7.8550 us/op	0.89
fastMsgIdFn h32 xxhash / 1000 bytes	352.00 ns/op	442.00 ns/op	0.80
fastMsgIdFn h64 xxhash / 1000 bytes	340.00 ns/op	379.00 ns/op	0.90
fastMsgIdFn sha256 / 10000 bytes	62.060 us/op	66.147 us/op	0.94
fastMsgIdFn h32 xxhash / 10000 bytes	1.7810 us/op	1.9820 us/op	0.90
fastMsgIdFn h64 xxhash / 10000 bytes	1.1830 us/op	1.3010 us/op	0.91
send data - 1000 256B messages	11.593 ms/op	14.029 ms/op	0.83
send data - 1000 512B messages	16.038 ms/op	18.425 ms/op	0.87
send data - 1000 1024B messages	25.283 ms/op	29.796 ms/op	0.85
send data - 1000 1200B messages	23.993 ms/op	26.579 ms/op	0.90
send data - 1000 2048B messages	29.900 ms/op	34.994 ms/op	0.85
send data - 1000 4096B messages	28.145 ms/op	32.319 ms/op	0.87
send data - 1000 16384B messages	68.150 ms/op	75.384 ms/op	0.90
send data - 1000 65536B messages	193.92 ms/op	239.27 ms/op	0.81
enrSubnets - fastDeserialize 64 bits	987.00 ns/op	1.1550 us/op	0.85
enrSubnets - ssz BitVector 64 bits	335.00 ns/op	405.00 ns/op	0.83
enrSubnets - fastDeserialize 4 bits	137.00 ns/op	192.00 ns/op	0.71
enrSubnets - ssz BitVector 4 bits	343.00 ns/op	450.00 ns/op	0.76
prioritizePeers score -10:0 att 32-0.1 sync 2-0	129.21 us/op	188.24 us/op	0.69
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	154.86 us/op	214.44 us/op	0.72
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	216.44 us/op	328.36 us/op	0.66
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	449.30 us/op	490.22 us/op	0.92
prioritizePeers score 0:0 att 64-1 sync 4-1	536.52 us/op	672.14 us/op	0.80
array of 16000 items push then shift	1.5974 us/op	1.7033 us/op	0.94
LinkedList of 16000 items push then shift	7.0420 ns/op	8.7420 ns/op	0.81
array of 16000 items push then pop	105.20 ns/op	140.65 ns/op	0.75
LinkedList of 16000 items push then pop	6.7510 ns/op	7.9340 ns/op	0.85
array of 24000 items push then shift	2.2973 us/op	2.4828 us/op	0.93
LinkedList of 24000 items push then shift	6.8910 ns/op	8.8330 ns/op	0.78
array of 24000 items push then pop	142.26 ns/op	164.64 ns/op	0.86
LinkedList of 24000 items push then pop	6.8110 ns/op	7.4090 ns/op	0.92
intersect bitArray bitLen 8	6.6240 ns/op	6.6620 ns/op	0.99
intersect array and set length 8	45.289 ns/op	46.657 ns/op	0.97
intersect bitArray bitLen 128	29.352 ns/op	31.228 ns/op	0.94
intersect array and set length 128	651.35 ns/op	691.96 ns/op	0.94
bitArray.getTrueBitIndexes() bitLen 128	1.3450 us/op	1.6090 us/op	0.84
bitArray.getTrueBitIndexes() bitLen 248	2.2980 us/op	2.5150 us/op	0.91
bitArray.getTrueBitIndexes() bitLen 512	4.2560 us/op	4.7850 us/op	0.89
Buffer.concat 32 items	887.00 ns/op	776.00 ns/op	1.14
Uint8Array.set 32 items	1.5740 us/op	1.4410 us/op	1.09
Buffer.copy	2.4680 us/op	2.6630 us/op	0.93
Uint8Array.set - with subarray	2.3940 us/op	2.2870 us/op	1.05
Uint8Array.set - without subarray	1.5580 us/op	1.4200 us/op	1.10
getUint32 - dataview	213.00 ns/op	233.00 ns/op	0.91
getUint32 - manual	143.00 ns/op	170.00 ns/op	0.84
Set add up to 64 items then delete first	2.1164 us/op	2.2593 us/op	0.94
OrderedSet add up to 64 items then delete first	3.1385 us/op	3.3249 us/op	0.94
Set add up to 64 items then delete last	2.4276 us/op	2.5328 us/op	0.96
OrderedSet add up to 64 items then delete last	3.5112 us/op	3.7826 us/op	0.93
Set add up to 64 items then delete middle	2.4599 us/op	2.5659 us/op	0.96
OrderedSet add up to 64 items then delete middle	5.0736 us/op	5.2497 us/op	0.97
Set add up to 128 items then delete first	4.9489 us/op	5.3500 us/op	0.93
OrderedSet add up to 128 items then delete first	7.6382 us/op	7.9665 us/op	0.96
Set add up to 128 items then delete last	4.8099 us/op	5.4551 us/op	0.88
OrderedSet add up to 128 items then delete last	6.9056 us/op	7.4776 us/op	0.92
Set add up to 128 items then delete middle	4.5628 us/op	5.3540 us/op	0.85
OrderedSet add up to 128 items then delete middle	12.899 us/op	15.682 us/op	0.82
Set add up to 256 items then delete first	9.6229 us/op	11.320 us/op	0.85
OrderedSet add up to 256 items then delete first	14.909 us/op	19.521 us/op	0.76
Set add up to 256 items then delete last	9.1300 us/op	12.036 us/op	0.76
OrderedSet add up to 256 items then delete last	13.814 us/op	18.509 us/op	0.75
Set add up to 256 items then delete middle	9.0052 us/op	11.724 us/op	0.77
OrderedSet add up to 256 items then delete middle	39.042 us/op	46.619 us/op	0.84
transfer serialized Status (84 B)	2.3450 us/op	2.8530 us/op	0.82
copy serialized Status (84 B)	1.1960 us/op	1.5070 us/op	0.79
transfer serialized SignedVoluntaryExit (112 B)	2.4550 us/op	2.9330 us/op	0.84
copy serialized SignedVoluntaryExit (112 B)	1.2600 us/op	1.4680 us/op	0.86
transfer serialized ProposerSlashing (416 B)	2.8580 us/op	3.1560 us/op	0.91
copy serialized ProposerSlashing (416 B)	1.7210 us/op	1.9010 us/op	0.91
transfer serialized Attestation (485 B)	2.7520 us/op	3.0880 us/op	0.89
copy serialized Attestation (485 B)	1.7240 us/op	1.8640 us/op	0.92
transfer serialized AttesterSlashing (33232 B)	2.9240 us/op	2.7640 us/op	1.06
copy serialized AttesterSlashing (33232 B)	4.4560 us/op	6.3920 us/op	0.70
transfer serialized Small SignedBeaconBlock (128000 B)	3.7730 us/op	3.6630 us/op	1.03
copy serialized Small SignedBeaconBlock (128000 B)	12.880 us/op	20.229 us/op	0.64
transfer serialized Avg SignedBeaconBlock (200000 B)	4.1680 us/op	4.5660 us/op	0.91
copy serialized Avg SignedBeaconBlock (200000 B)	19.129 us/op	27.684 us/op	0.69
transfer serialized BlobsSidecar (524380 B)	4.1030 us/op	4.7600 us/op	0.86
copy serialized BlobsSidecar (524380 B)	80.787 us/op	92.429 us/op	0.87
transfer serialized Big SignedBeaconBlock (1000000 B)	3.8990 us/op	5.3240 us/op	0.73
copy serialized Big SignedBeaconBlock (1000000 B)	149.78 us/op	396.50 us/op	0.38
pass gossip attestations to forkchoice per slot	2.7462 ms/op	2.9228 ms/op	0.94
forkChoice updateHead vc 100000 bc 64 eq 0	486.94 us/op	473.46 us/op	1.03
forkChoice updateHead vc 600000 bc 64 eq 0	2.7195 ms/op	3.1543 ms/op	0.86
forkChoice updateHead vc 1000000 bc 64 eq 0	4.8024 ms/op	5.1736 ms/op	0.93
forkChoice updateHead vc 600000 bc 320 eq 0	2.7586 ms/op	3.0562 ms/op	0.90
forkChoice updateHead vc 600000 bc 1200 eq 0	2.8443 ms/op	2.9986 ms/op	0.95
forkChoice updateHead vc 600000 bc 7200 eq 0	3.8870 ms/op	3.8554 ms/op	1.01
forkChoice updateHead vc 600000 bc 64 eq 1000	10.255 ms/op	10.649 ms/op	0.96
forkChoice updateHead vc 600000 bc 64 eq 10000	10.198 ms/op	10.644 ms/op	0.96
forkChoice updateHead vc 600000 bc 64 eq 300000	13.716 ms/op	15.971 ms/op	0.86
computeDeltas 500000 validators 300 proto nodes	3.9637 ms/op	4.2893 ms/op	0.92
computeDeltas 500000 validators 1200 proto nodes	4.0380 ms/op	4.1697 ms/op	0.97
computeDeltas 500000 validators 7200 proto nodes	3.9211 ms/op	4.1352 ms/op	0.95
computeDeltas 750000 validators 300 proto nodes	5.7207 ms/op	6.3695 ms/op	0.90
computeDeltas 750000 validators 1200 proto nodes	5.8779 ms/op	6.2209 ms/op	0.94
computeDeltas 750000 validators 7200 proto nodes	5.8971 ms/op	6.4931 ms/op	0.91
computeDeltas 1400000 validators 300 proto nodes	11.214 ms/op	11.953 ms/op	0.94
computeDeltas 1400000 validators 1200 proto nodes	11.196 ms/op	11.759 ms/op	0.95
computeDeltas 1400000 validators 7200 proto nodes	11.080 ms/op	11.898 ms/op	0.93
computeDeltas 2100000 validators 300 proto nodes	16.198 ms/op	18.269 ms/op	0.89
computeDeltas 2100000 validators 1200 proto nodes	16.146 ms/op	18.629 ms/op	0.87
computeDeltas 2100000 validators 7200 proto nodes	16.111 ms/op	17.742 ms/op	0.91
altair processAttestation - 250000 vs - 7PWei normalcase	1.8069 ms/op	2.1543 ms/op	0.84
altair processAttestation - 250000 vs - 7PWei worstcase	2.5625 ms/op	3.5250 ms/op	0.73
altair processAttestation - setStatus - 1/6 committees join	88.372 us/op	140.13 us/op	0.63
altair processAttestation - setStatus - 1/3 committees join	179.77 us/op	244.36 us/op	0.74
altair processAttestation - setStatus - 1/2 committees join	256.77 us/op	332.28 us/op	0.77
altair processAttestation - setStatus - 2/3 committees join	331.76 us/op	455.09 us/op	0.73
altair processAttestation - setStatus - 4/5 committees join	472.87 us/op	612.13 us/op	0.77
altair processAttestation - setStatus - 100% committees join	561.87 us/op	743.21 us/op	0.76
altair processBlock - 250000 vs - 7PWei normalcase	3.6712 ms/op	4.9100 ms/op	0.75
altair processBlock - 250000 vs - 7PWei normalcase hashState	25.598 ms/op	31.975 ms/op	0.80
altair processBlock - 250000 vs - 7PWei worstcase	33.077 ms/op	42.483 ms/op	0.78
altair processBlock - 250000 vs - 7PWei worstcase hashState	75.736 ms/op	78.945 ms/op	0.96
phase0 processBlock - 250000 vs - 7PWei normalcase	1.5268 ms/op	2.8450 ms/op	0.54
phase0 processBlock - 250000 vs - 7PWei worstcase	19.285 ms/op	29.773 ms/op	0.65
altair processEth1Data - 250000 vs - 7PWei normalcase	293.32 us/op	329.65 us/op	0.89
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15	6.4050 us/op	9.3670 us/op	0.68
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219	30.572 us/op	72.768 us/op	0.42
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42	8.6420 us/op	19.032 us/op	0.45
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18	6.6290 us/op	10.249 us/op	0.65
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020	143.99 us/op	224.53 us/op	0.64
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777	1.2781 ms/op	1.2349 ms/op	1.03
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384	1.7834 ms/op	2.3140 ms/op	0.77
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	1.4858 ms/op	1.8216 ms/op	0.82
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	4.1794 ms/op	5.1659 ms/op	0.81
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384	1.7319 ms/op	2.0149 ms/op	0.86
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	3.9000 ms/op	4.7926 ms/op	0.81
Tree 40 250000 create	207.00 ms/op	253.23 ms/op	0.82
Tree 40 250000 get(125000)	138.64 ns/op	162.43 ns/op	0.85
Tree 40 250000 set(125000)	610.90 ns/op	741.34 ns/op	0.82
Tree 40 250000 toArray()	14.676 ms/op	21.468 ms/op	0.68
Tree 40 250000 iterate all - toArray() + loop	15.045 ms/op	21.387 ms/op	0.70
Tree 40 250000 iterate all - get(i)	49.935 ms/op	59.088 ms/op	0.85
Array 250000 create	2.7495 ms/op	3.9176 ms/op	0.70
Array 250000 clone - spread	1.3951 ms/op	1.6164 ms/op	0.86
Array 250000 get(125000)	0.40000 ns/op	0.47500 ns/op	0.84
Array 250000 set(125000)	0.41800 ns/op	0.51400 ns/op	0.81
Array 250000 iterate all - loop	79.400 us/op	94.157 us/op	0.84
phase0 afterProcessEpoch - 250000 vs - 7PWei	48.228 ms/op	57.027 ms/op	0.85
Array.fill - length 1000000	3.5888 ms/op	5.5505 ms/op	0.65
Array push - length 1000000	16.267 ms/op	21.109 ms/op	0.77
Array.get	0.28059 ns/op	0.29776 ns/op	0.94
Uint8Array.get	0.44300 ns/op	0.47685 ns/op	0.93
phase0 beforeProcessEpoch - 250000 vs - 7PWei	17.203 ms/op	20.567 ms/op	0.84
altair processEpoch - mainnet_e81889	263.18 ms/op	317.86 ms/op	0.83
mainnet_e81889 - altair beforeProcessEpoch	16.512 ms/op	21.073 ms/op	0.78
mainnet_e81889 - altair processJustificationAndFinalization	13.316 us/op	17.781 us/op	0.75
mainnet_e81889 - altair processInactivityUpdates	4.8195 ms/op	7.1549 ms/op	0.67
mainnet_e81889 - altair processRewardsAndPenalties	41.626 ms/op	44.795 ms/op	0.93
mainnet_e81889 - altair processRegistryUpdates	2.3380 us/op	5.2460 us/op	0.45
mainnet_e81889 - altair processSlashings	484.00 ns/op	594.00 ns/op	0.81
mainnet_e81889 - altair processEth1DataReset	501.00 ns/op	555.00 ns/op	0.90
mainnet_e81889 - altair processEffectiveBalanceUpdates	1.3645 ms/op	1.7784 ms/op	0.77
mainnet_e81889 - altair processSlashingsReset	4.4520 us/op	8.3160 us/op	0.54
mainnet_e81889 - altair processRandaoMixesReset	4.7740 us/op	7.8260 us/op	0.61
mainnet_e81889 - altair processHistoricalRootsUpdate	695.00 ns/op	895.00 ns/op	0.78
mainnet_e81889 - altair processParticipationFlagUpdates	1.8770 us/op	3.2410 us/op	0.58
mainnet_e81889 - altair processSyncCommitteeUpdates	742.00 ns/op	1.3310 us/op	0.56
mainnet_e81889 - altair afterProcessEpoch	49.035 ms/op	52.158 ms/op	0.94
capella processEpoch - mainnet_e217614	939.42 ms/op	1.2015 s/op	0.78
mainnet_e217614 - capella beforeProcessEpoch	70.375 ms/op	85.113 ms/op	0.83
mainnet_e217614 - capella processJustificationAndFinalization	13.011 us/op	17.889 us/op	0.73
mainnet_e217614 - capella processInactivityUpdates	16.350 ms/op	19.372 ms/op	0.84
mainnet_e217614 - capella processRewardsAndPenalties	208.06 ms/op	237.64 ms/op	0.88
mainnet_e217614 - capella processRegistryUpdates	18.365 us/op	19.840 us/op	0.93
mainnet_e217614 - capella processSlashings	491.00 ns/op	727.00 ns/op	0.68
mainnet_e217614 - capella processEth1DataReset	386.00 ns/op	1.4760 us/op	0.26
mainnet_e217614 - capella processEffectiveBalanceUpdates	11.556 ms/op	17.574 ms/op	0.66
mainnet_e217614 - capella processSlashingsReset	4.1520 us/op	6.0130 us/op	0.69
mainnet_e217614 - capella processRandaoMixesReset	4.5820 us/op	6.6620 us/op	0.69
mainnet_e217614 - capella processHistoricalRootsUpdate	494.00 ns/op	782.00 ns/op	0.63
mainnet_e217614 - capella processParticipationFlagUpdates	1.9790 us/op	2.7930 us/op	0.71
mainnet_e217614 - capella afterProcessEpoch	123.75 ms/op	127.13 ms/op	0.97
phase0 processEpoch - mainnet_e58758	253.07 ms/op	337.48 ms/op	0.75
mainnet_e58758 - phase0 beforeProcessEpoch	66.755 ms/op	96.191 ms/op	0.69
mainnet_e58758 - phase0 processJustificationAndFinalization	11.374 us/op	17.203 us/op	0.66
mainnet_e58758 - phase0 processRewardsAndPenalties	28.544 ms/op	27.590 ms/op	1.03
mainnet_e58758 - phase0 processRegistryUpdates	10.911 us/op	14.619 us/op	0.75
mainnet_e58758 - phase0 processSlashings	483.00 ns/op	652.00 ns/op	0.74
mainnet_e58758 - phase0 processEth1DataReset	421.00 ns/op	512.00 ns/op	0.82
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.1647 ms/op	1.4284 ms/op	0.82
mainnet_e58758 - phase0 processSlashingsReset	2.8630 us/op	2.7580 us/op	1.04
mainnet_e58758 - phase0 processRandaoMixesReset	4.3670 us/op	5.3470 us/op	0.82
mainnet_e58758 - phase0 processHistoricalRootsUpdate	395.00 ns/op	649.00 ns/op	0.61
mainnet_e58758 - phase0 processParticipationRecordUpdates	4.0630 us/op	5.1810 us/op	0.78
mainnet_e58758 - phase0 afterProcessEpoch	43.450 ms/op	43.762 ms/op	0.99
phase0 processEffectiveBalanceUpdates - 250000 normalcase	1.3343 ms/op	1.4995 ms/op	0.89
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	1.9390 ms/op	2.0602 ms/op	0.94
altair processInactivityUpdates - 250000 normalcase	14.525 ms/op	21.428 ms/op	0.68
altair processInactivityUpdates - 250000 worstcase	14.000 ms/op	20.592 ms/op	0.68
phase0 processRegistryUpdates - 250000 normalcase	10.230 us/op	11.245 us/op	0.91
phase0 processRegistryUpdates - 250000 badcase_full_deposits	310.20 us/op	526.10 us/op	0.59
phase0 processRegistryUpdates - 250000 worstcase 0.5	103.12 ms/op	113.99 ms/op	0.90
altair processRewardsAndPenalties - 250000 normalcase	37.014 ms/op	43.912 ms/op	0.84
altair processRewardsAndPenalties - 250000 worstcase	37.808 ms/op	46.526 ms/op	0.81
phase0 getAttestationDeltas - 250000 normalcase	9.6840 ms/op	7.6087 ms/op	1.27
phase0 getAttestationDeltas - 250000 worstcase	8.1478 ms/op	7.7062 ms/op	1.06
phase0 processSlashings - 250000 worstcase	102.42 us/op	136.50 us/op	0.75
altair processSyncCommitteeUpdates - 250000	131.50 ms/op	121.28 ms/op	1.08
BeaconState.hashTreeRoot - No change	249.00 ns/op	234.00 ns/op	1.06
BeaconState.hashTreeRoot - 1 full validator	95.258 us/op	110.52 us/op	0.86
BeaconState.hashTreeRoot - 32 full validator	920.06 us/op	1.2236 ms/op	0.75
BeaconState.hashTreeRoot - 512 full validator	9.7222 ms/op	11.179 ms/op	0.87
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	100.89 us/op	137.93 us/op	0.73
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	1.5903 ms/op	2.0293 ms/op	0.78
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	19.032 ms/op	23.071 ms/op	0.82
BeaconState.hashTreeRoot - 1 balances	86.566 us/op	98.803 us/op	0.88
BeaconState.hashTreeRoot - 32 balances	822.58 us/op	1.0152 ms/op	0.81
BeaconState.hashTreeRoot - 512 balances	7.5730 ms/op	8.1660 ms/op	0.93
BeaconState.hashTreeRoot - 250000 balances	189.65 ms/op	153.72 ms/op	1.23
aggregationBits - 2048 els - zipIndexesInBitList	27.354 us/op	23.814 us/op	1.15
byteArrayEquals 32	54.284 ns/op	53.734 ns/op	1.01
Buffer.compare 32	17.446 ns/op	17.297 ns/op	1.01
byteArrayEquals 1024	1.6053 us/op	1.5885 us/op	1.01
Buffer.compare 1024	26.068 ns/op	25.763 ns/op	1.01
byteArrayEquals 16384	25.587 us/op	25.277 us/op	1.01
Buffer.compare 16384	233.22 ns/op	193.58 ns/op	1.20
byteArrayEquals 123687377	192.42 ms/op	191.38 ms/op	1.01
Buffer.compare 123687377	7.3462 ms/op	6.3346 ms/op	1.16
byteArrayEquals 32 - diff last byte	52.333 ns/op	52.755 ns/op	0.99
Buffer.compare 32 - diff last byte	17.111 ns/op	17.317 ns/op	0.99
byteArrayEquals 1024 - diff last byte	1.5812 us/op	1.5925 us/op	0.99
Buffer.compare 1024 - diff last byte	25.064 ns/op	26.238 ns/op	0.96
byteArrayEquals 16384 - diff last byte	25.315 us/op	25.326 us/op	1.00
Buffer.compare 16384 - diff last byte	209.39 ns/op	204.50 ns/op	1.02
byteArrayEquals 123687377 - diff last byte	190.41 ms/op	194.62 ms/op	0.98
Buffer.compare 123687377 - diff last byte	7.3179 ms/op	6.9820 ms/op	1.05
byteArrayEquals 32 - random bytes	5.1460 ns/op	5.3490 ns/op	0.96
Buffer.compare 32 - random bytes	17.618 ns/op	17.843 ns/op	0.99
byteArrayEquals 1024 - random bytes	5.1430 ns/op	5.3130 ns/op	0.97
Buffer.compare 1024 - random bytes	17.506 ns/op	17.882 ns/op	0.98
byteArrayEquals 16384 - random bytes	5.1240 ns/op	5.3010 ns/op	0.97
Buffer.compare 16384 - random bytes	17.722 ns/op	17.867 ns/op	0.99
byteArrayEquals 123687377 - random bytes	6.4600 ns/op	6.7000 ns/op	0.96
Buffer.compare 123687377 - random bytes	18.880 ns/op	18.990 ns/op	0.99
regular array get 100000 times	32.805 us/op	35.848 us/op	0.92
wrappedArray get 100000 times	32.886 us/op	34.067 us/op	0.97
arrayWithProxy get 100000 times	13.011 ms/op	13.229 ms/op	0.98
ssz.Root.equals	45.820 ns/op	47.258 ns/op	0.97
byteArrayEquals	45.129 ns/op	46.529 ns/op	0.97
Buffer.compare	10.442 ns/op	10.802 ns/op	0.97
processSlot - 1 slots	15.105 us/op	13.402 us/op	1.13
processSlot - 32 slots	2.4965 ms/op	3.0955 ms/op	0.81
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	35.396 ms/op	38.550 ms/op	0.92
getCommitteeAssignments - req 1 vs - 250000 vc	2.0583 ms/op	2.2617 ms/op	0.91
getCommitteeAssignments - req 100 vs - 250000 vc	4.0165 ms/op	4.2529 ms/op	0.94
getCommitteeAssignments - req 1000 vs - 250000 vc	4.2891 ms/op	4.5708 ms/op	0.94
findModifiedValidators - 10000 modified validators	271.07 ms/op	226.25 ms/op	1.20
findModifiedValidators - 1000 modified validators	181.97 ms/op	180.35 ms/op	1.01
findModifiedValidators - 100 modified validators	197.29 ms/op	168.90 ms/op	1.17
findModifiedValidators - 10 modified validators	173.44 ms/op	172.22 ms/op	1.01
findModifiedValidators - 1 modified validators	166.00 ms/op	171.21 ms/op	0.97
findModifiedValidators - no difference	218.64 ms/op	171.46 ms/op	1.28
compare ViewDUs	3.2025 s/op	3.6014 s/op	0.89
compare each validator Uint8Array	1.5603 s/op	1.3366 s/op	1.17
compare ViewDU to Uint8Array	976.23 ms/op	1.2270 s/op	0.80
migrate state 1000000 validators, 24 modified, 0 new	715.49 ms/op	870.02 ms/op	0.82
migrate state 1000000 validators, 1700 modified, 1000 new	1.0028 s/op	1.0390 s/op	0.97
migrate state 1000000 validators, 3400 modified, 2000 new	1.1467 s/op	1.2435 s/op	0.92
migrate state 1500000 validators, 24 modified, 0 new	717.01 ms/op	870.07 ms/op	0.82
migrate state 1500000 validators, 1700 modified, 1000 new	948.22 ms/op	1.1091 s/op	0.85
migrate state 1500000 validators, 3400 modified, 2000 new	1.1600 s/op	1.4065 s/op	0.82
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	4.2200 ns/op	5.1800 ns/op	0.81
state getBlockRootAtSlot - 250000 vs - 7PWei	813.05 ns/op	722.48 ns/op	1.13
computeProposers - vc 250000	7.0751 ms/op	8.3247 ms/op	0.85
computeEpochShuffling - vc 250000	39.930 ms/op	44.264 ms/op	0.90
getNextSyncCommittee - vc 250000	126.31 ms/op	149.46 ms/op	0.85
computeSigningRoot for AttestationData	25.176 us/op	26.738 us/op	0.94
hash AttestationData serialized data then Buffer.toString(base64)	1.4971 us/op	1.6957 us/op	0.88
toHexString serialized data	829.27 ns/op	985.91 ns/op	0.84
Buffer.toString(base64)	165.29 ns/op	196.09 ns/op	0.84
nodejs block root to RootHex using toHex	143.62 ns/op	178.10 ns/op	0.81
nodejs block root to RootHex using toRootHex	83.760 ns/op	114.49 ns/op	0.73
browser block root to RootHex using the deprecated toHexString	220.12 ns/op	273.26 ns/op	0.81
browser block root to RootHex using toHex	176.44 ns/op	230.82 ns/op	0.76
browser block root to RootHex using toRootHex	156.43 ns/op	176.82 ns/op	0.88

by benchmarkbot/action

[nflaig]

LGTM

[nflaig]

Steps to test or reproduce
resync mekong?

This will be hard to reproduce, isn't the only way to trigger this issue if we run the epoch transition twice, ie. if there is a reorg across epoch boundaries?

From looking at mekong nodes, there is just a single node (lodestar-geth-3) that has this issue.