bug: Event indexing reorg protection doesn't work if transaction gets reorged into different

[fleupold]

Problem

The other day we saw our competition endpoint stop showing data for recent transactions. This was due to our settlement event indexing code failing because the same settlement transaction showed up in two different blocks according to our database. The block prior to the one etherscan shows now had been re-orged, our settlement had been removed from it and was included in the next block instead. In the database the same settlement tx hash then showed up associated with two different blocks (settlements table) causing a subquery to fail since there was more than one row to return.

Impact

Settlement indexing which is used for debugging and rewards calculation stopped, record had to be manually removed.

To reproduce

Write an integration test for the autopilot that uses anvil to simulate reorg behavior.

• Mine settlement in block x
• revert chain to block x-1
• mine a block without settlement (block x)
• mine settlement in block x+1

Expected behaviour

Reorg detection deletes the first settlement in favor of the second settlement

Screenshots/logs

https://production-6de61f.kb.eu-central-1.aws.cloud.es.io/app/r/s/GKc86 l

Additional context

I believe the offending code is here:

services/crates/shared/src/event_handling.rs

Lines 336 to 357 in 148a922

	async fn update_events_from_latest_blocks(
	&mut self,
	latest_blocks: &[BlockNumberHash],
	is_reorg: bool,
	) -> Result<()> {
	debug_assert!(
	!latest_blocks.is_empty(),
	"entered update events with empty block list"
	);
	let (blocks, events) = self.past_events_by_block_hashes(latest_blocks).await;
	track_block_range(&format!("range_{}", blocks.len()));
	if blocks.is_empty() {
	return Err(anyhow::anyhow!(
	"no blocks to be updated - all filtered out"
	));
	}
	// update storage regardless if it's a full update or partial update
	let range = RangeInclusive::try_new(blocks.first().unwrap().0, blocks.last().unwrap().0)?;
	if is_reorg {
	self.store.replace_events(events, range.clone()).await?;
	} else {

We replace events in the range that was returned by self.past_events_by_block_hashes. This however only returns the blocks that contain events according to the reorged chain (in the example above block x+1). The previous settlement is stored in block x however. This means we don't replace the previous event and only add the new event, unless the settlement has been reorged to the same block (if reorged to a prior block I believe we will also miss it, but this doesn't really happen in practice).

I believe the fix would be to replace events on the old block range (latest_blocks.first() -> latest_blocks.last()). This may cause issues with partial updates, which I'm not sure are used in practice. Better tests of this code is needed to clarify.

[github-actions]

This issue has been marked as stale because it has been inactive a while. Please update this issue or it will be automatically closed.

[fleupold]

Event indexing code is being reworked and this is likely no longer relevant