[Persistence] Adds Save and Load functionality to TreeStore #897

dylanlott · 2023-07-11T21:40:24Z

Description

Adds save and load functionality to the TreeStore.

This PR makes the TreeStore an Interruptable module. This is necessary because the Persistence module needs to be able to tell the TreeStore to relinquish its connection to the nodeStores which are backed by BadgerDB.

Issue

Related to #568

Type of change

Please mark the relevant option(s):

List of changes

Adds save and load functionality for loading a worldState from file directories.

Testing

make develop_test; if any code changes were made
make test_e2e on k8s LocalNet; if any code changes were made
e2e-devnet-test passes tests on DevNet; if any code was changed
Docker Compose LocalNet; if any major functionality was changed or introduced
k8s LocalNet; if any infrastructure or configuration changes were made

Required Checklist

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have added, or updated, godoc format comments on touched members (see: tip.golang.org/doc/comment)
I have tested my changes using the available tooling
I have updated the corresponding CHANGELOG

If Applicable Checklist

I have updated the corresponding README(s); local and/or global
I have added tests that prove my fix is effective or that my feature works
I have added, or updated, mermaid.js diagrams in the corresponding README(s)
I have added, or updated, documentation and mermaid.js diagrams in shared/docs/* if I updated shared/*README(s)

* fulfills the Interruptable module interface to allow for control over the node stores lifecycle to enable save and load functionality * adds tests for the Start and Stop functionality * adds nil checks to GetTree function to make testing start and stop behavior predictable

Olshansk

A few comments throughout but nothing "glaring" or blocking.

Thanks for getting this moving Dylan!

persistence/blockstore/block_store.go

persistence/kvstore/kvstore.go

persistence/trees/trees_test.go

persistence/kvstore/kvstore_test.go

Olshansk · 2023-08-01T22:44:46Z

persistence/kvstore/kvstore_test.go

 func setupStore(t *testing.T, store KVStore) {
 	t.Helper()
 	err := store.Set([]byte("foo"), []byte("bar"))
 	require.NoError(t, err)
 	err = store.Set([]byte("baz"), []byte("bin"))
 	require.NoError(t, err)
 }
+
+func isEmpty(t *testing.T, dir string) (bool, error) {


Why not go with an approach like this: https://stackoverflow.com/a/24102536/768439 ?

Seems shorter

Beacuse the linter complains if ioutil is used since it's officially deprecated.

why not use

dirEntries, err := os.ReadDir(dir) require.NoError(t, err) if len(dirEntries) == 0 { return true } return false

https://pkg.go.dev/os#ReadDir

This is what ioutil docs point to as the newer more efficient approach to the deprecated method

persistence/trees/trees.go

Olshansk · 2023-08-04T20:55:34Z

@dylanlott Can you "re-request" a review when its ready and I'll reply to all the comments/questions then?

dylanlott · 2023-08-04T21:03:47Z

@dylanlott Can you "re-request" a review when its ready and I'll reply to all the comments/questions then?

You beat me to it. I just finished up addressing your feedback, it's ready for re-review 👍

reviewpad bot added the large Pull request is large label Jul 11, 2023

dylanlott self-assigned this Jul 11, 2023

dylanlott added the persistence Persistence specific changes label Jul 11, 2023

dylanlott changed the base branch from main to persistence/savepoints-initial July 11, 2023 21:45

dylanlott force-pushed the persistence/save-load branch from f2ac214 to 9b68e36 Compare July 11, 2023 21:46

reviewpad bot added medium Pull request is medium and removed large Pull request is large labels Jul 11, 2023

dylanlott changed the title ~~Adds Save and Load functionality to TreeStore~~ [Persistence] Adds Save and Load functionality to TreeStore Jul 11, 2023

dylanlott force-pushed the persistence/savepoints-initial branch from 0a6c72f to 7de2528 Compare July 12, 2023 15:12

dylanlott changed the title ~~[Persistence] Adds Save and Load functionality to TreeStore~~ [WIP][Persistence] Adds Save and Load functionality to TreeStore Jul 12, 2023

reviewpad bot added large Pull request is large and removed medium Pull request is medium labels Jul 12, 2023

dylanlott force-pushed the persistence/savepoints-initial branch 2 times, most recently from 2d53dc1 to 15d5c8a Compare July 17, 2023 19:15

dylanlott force-pushed the persistence/save-load branch from 9b68e36 to 99d12f4 Compare July 17, 2023 20:02

reviewpad bot added medium Pull request is medium and removed large Pull request is large labels Jul 17, 2023

dylanlott force-pushed the persistence/savepoints-initial branch from 15d5c8a to a6fe96f Compare July 18, 2023 20:37

dylanlott force-pushed the persistence/save-load branch from 99d12f4 to 7a271fe Compare July 18, 2023 21:04

dylanlott mentioned this pull request Jul 19, 2023

[Persistence] Adds atomic Update for TreeStore #861

Merged

20 tasks

dylanlott force-pushed the persistence/save-load branch from 7a271fe to fb904c9 Compare July 19, 2023 22:59

dylanlott force-pushed the persistence/savepoints-initial branch from 788dbbe to c9caaa3 Compare July 20, 2023 00:27

dylanlott force-pushed the persistence/save-load branch from fb904c9 to e6337f8 Compare July 20, 2023 00:28

dylanlott force-pushed the persistence/savepoints-initial branch 2 times, most recently from 9de1b01 to dc4f205 Compare July 20, 2023 17:08

dylanlott force-pushed the persistence/save-load branch from e6337f8 to d27ee09 Compare July 20, 2023 17:09

reviewpad bot added large Pull request is large and removed medium Pull request is medium labels Jul 20, 2023

dylanlott force-pushed the persistence/savepoints-initial branch 2 times, most recently from 5b7af11 to a1e0a10 Compare July 24, 2023 18:15

dylanlott force-pushed the persistence/save-load branch 2 times, most recently from 8b4844c to 4765b3a Compare July 26, 2023 17:04

dylanlott force-pushed the persistence/savepoints-initial branch 2 times, most recently from 29ad7df to 0f87f0b Compare July 27, 2023 17:24

dylanlott force-pushed the persistence/save-load branch from 4765b3a to d6550f1 Compare July 29, 2023 01:26

Base automatically changed from persistence/savepoints-initial to main July 31, 2023 16:04

dylanlott force-pushed the persistence/save-load branch from ec24322 to 0c6e894 Compare July 31, 2023 20:37

dylanlott added 3 commits July 31, 2023 19:36

[WIP] initial commit gets save and load working

0d34af1

[wip] gets destination backup working for KVStore

2c6fb5c

dylanlott force-pushed the persistence/save-load branch from 7f3d341 to 2c6fb5c Compare August 1, 2023 02:19

[wip] removes TreeStore Start call from Create

14e4225

dylanlott changed the title ~~[WIP][Persistence] Adds Save and Load functionality to TreeStore~~ [Persistence] Adds Save and Load functionality to TreeStore Aug 1, 2023

dylanlott requested review from Olshansk and h5law August 1, 2023 16:09

dylanlott added e2e-devnet-test Runs E2E tests on devnet waiting-for-review labels Aug 1, 2023

[fixup] remove unnecessary fmt.Printf

a56c2cd

dylanlott marked this pull request as ready for review August 1, 2023 21:27

Olshansk requested changes Aug 2, 2023

View reviewed changes

dylanlott added 2 commits August 2, 2023 11:34

[cleanup] addressing review feedback

0a608f8

Merge branch 'main' into persistence/save-load

df52d32

[fixup] addressing review feedback

9f05e2d

dylanlott requested a review from Olshansk August 4, 2023 21:03

dylanlott added 2 commits August 4, 2023 15:05

[fixup] remove comment

bd48902

Merge remote-tracking branch 'origin' into persistence/save-load

fb3adb8

h5law removed their request for review December 10, 2023 20:38

dylanlott closed this Jul 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Persistence] Adds Save and Load functionality to TreeStore #897

[Persistence] Adds Save and Load functionality to TreeStore #897

dylanlott commented Jul 11, 2023 •

edited

Loading

Olshansk left a comment

Olshansk Aug 1, 2023

dylanlott Aug 2, 2023

h5law Aug 9, 2023 •

edited

Loading

Olshansk commented Aug 4, 2023

dylanlott commented Aug 4, 2023

[Persistence] Adds Save and Load functionality to TreeStore #897

[Persistence] Adds Save and Load functionality to TreeStore #897

Conversation

dylanlott commented Jul 11, 2023 • edited Loading

Description

Issue

Type of change

List of changes

Testing

Required Checklist

If Applicable Checklist

Olshansk left a comment

Choose a reason for hiding this comment

Olshansk Aug 1, 2023

Choose a reason for hiding this comment

dylanlott Aug 2, 2023

Choose a reason for hiding this comment

h5law Aug 9, 2023 • edited Loading

Choose a reason for hiding this comment

Olshansk commented Aug 4, 2023

dylanlott commented Aug 4, 2023

dylanlott commented Jul 11, 2023 •

edited

Loading

h5law Aug 9, 2023 •

edited

Loading