Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1850 historic node reset db #1854

Merged
merged 20 commits into from
Apr 8, 2024
Merged

Conversation

kladkogex
Copy link
Collaborator

@kladkogex kladkogex commented Mar 20, 2024

The problem was that MANIFEST file of historic state LevelDB was growing indefinitely.

Since during restart MANIFEST file is read and processed, it led to extremely long opening times.

  1. A new config parameter has been added

leveldbReopenIntervalMs

The default value of leveldbReopenIntervalMs is one day or 24 * 60 * 60 * 1000 microseconds.

If leveldbReopenIntervalMs is negative (like -1) than the features is disabled

  1. Everytime there is a write batch to LevelDB (this typically happens after block processing) there is a check for the
    age of LevelDB descriptor. If the descriptor age is more than leveldbReopenIntervalMs, the descriptor is closed and opened again.

  2. Reopen of the descriptor happens while holding write lock (inclusive lock) Use of the descriptor happens while holding read lock.

  3. A new test infinite_loop_test is used to test the functionality. During the test, transactions are generated in the blockchain, while leveldbReopenIntervalMsis set to 1ms. Manifest file of leveldb is observed to see that it does not grow.

  4. Also readability of github tests were improved by adding --report_level=detailed options to testheth.
    Now if one of test_eth tests fails, it is way faster to understand which test fails and get the error message

@kladkogex kladkogex added this to the SKALE 2.3.1 (Archive node) milestone Mar 20, 2024
@kladkogex kladkogex linked an issue Mar 20, 2024 that may be closed by this pull request
@DmytroNazarenko DmytroNazarenko merged commit b7ce5d4 into v3.17.2 Apr 8, 2024
11 checks passed
@DmytroNazarenko DmytroNazarenko deleted the 1850_historic_node_reset_db branch April 8, 2024 14:19
@github-actions github-actions bot locked and limited conversation to collaborators Apr 8, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Historic node needs to close and reopen leveldb every day
3 participants