Skip to content
This repository has been archived by the owner on Dec 10, 2024. It is now read-only.

LatestTokenBalance init redesign #536

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

LatestTokenBalance init redesign #536

wants to merge 8 commits into from

Conversation

nemo83
Copy link
Contributor

@nemo83 nemo83 commented Aug 21, 2024

Most of the Schedules jobs run okay when they are working in an incremental way, meaning that they are close to tip and must update or aggreagate only a small amount of data.

Some jobs also have an "init" function, to process large amount of data when they run for the first time.

This PR aims to improve some of these jobs.

Some features"

  • added params to enable/disable all scheduled jobs. set false by default in dev
  • replaced init incremental sql approach for LatestTokenBalance w/ insert/select query. While this query runs in under 1 minutes in preprod/preview, on mainnet it takes several hours, but it runs entirely on db, w/o moving data back and forth from db to java and back to db.
  • changed the logic of the first round of cleanup for address balance. The first time (and until a flag is persisted in redis!) the address cleanup job starts, will purge address_balance history in slots interval. This is NOT a comprehensive cleanup, but aims to quickly throw away balances for addresses that are change very very frenquently. First complete run takes more than a day, but it's really lightweight on the db as it hits many indexes and small slot interval.

@nemo83 nemo83 changed the base branch from 1.7.0 to main August 21, 2024 11:59
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant