fixed: flush remaining rows on termination #67
Conversation
|
|
||
| s.OnTerminating(func(_ error) { s.stats.Close() }) |
There was a problem hiding this comment.
It's when s terminates that we want to close the stats. It feels weird that stats is registering its own Terminating handler to close itself.
| s.lastSeenCursor = cursor | ||
| s.lastSeenFinalBlock = data.FinalBlockHeight |
There was a problem hiding this comment.
Should we simply add those to flush directly instead? That would avoid maintaining some state before calling the function.
It also makes me wondering what happen if flush is called through OnTerminating but last block cursor/final block is the one before. It seems that it's a possibility here.
There was a problem hiding this comment.
Should we simply add those to flush directly instead? That would avoid maintaining some state before calling the function.
That doesn't work because we don't know the last seen cursor and final block when executing flush from OnTerminating
It also makes me wondering what happen if flush is called through OnTerminating but last block cursor/final block is the one before. It seems that it's a possibility here.
True, I guess we can just lock a mutex in HandleBlockScopedData to ensure that we always have a block fully processed before anything is flushed.
this fixes a potential case of missing block data in case the stop block number doesn't match the flush interval. So if you were processing the range of 0:1100 with a flush interval of 1000 you'd be missing all blocks > 1000 as flushes only happen on
%1000block numbers.The sql-sink will automatically stop at 1100, not flush the block range of 1000 to 1100 and also not produce any error letting the user know that there are blocks within their requested ranges that haven't been flushed.