Design solution for automated backfills

[deniseli]

A backfill is a retroactive update to historical data. i.e. modifying the existing rows of a table, or inserting missed rows after they're discovered to be missing. Common use cases:

• You ran a DB migration changing the schema of a table from having 1 big JSON blob column to 3 separate enum columns. Immediately upon changing your table's schema, you will have these 3 new columns, but they will be null for all the existing rows. You can backfill the new columns in all these old rows to make sure the historical data is compatible with your latest DAL+downstream code.
• Your system collects a bunch of raw telemetry data and then processes those logs into a more consumable log format using some ETL. Your ETL pipeline goes down for a day. After the outage is over, there is a full day's worth of raw telemetry data that has yet to be processed into your consumable log format. You could backfill the consumable table (i.e. re-run the ETL over the historical logs, bound to the exact time range of the outage) to fix the gap in your consumable data.

[alecthomas]

The async call system would be ideal for implementing this.

[alecthomas]

Backfills could be a fantastic first test case for "eating our own dogfood", ie. implementing backfills in a system FTL module.