Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: follower node sync from DA #631

Closed
wants to merge 73 commits into from
Closed
Show file tree
Hide file tree
Changes from 69 commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
4fe6c3a
implement l1rpcfetcher and prepare for implementing da_syncer
NazariiDenha Feb 5, 2024
25bc860
finish da_syncer, add db access, small fixes
NazariiDenha Feb 13, 2024
deed82a
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
NazariiDenha Feb 13, 2024
635f744
fix lint
NazariiDenha Feb 13, 2024
69d6c39
address comments
NazariiDenha Feb 19, 2024
8bc5a6b
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
NazariiDenha Mar 18, 2024
2bef690
poc
NazariiDenha Apr 9, 2024
b6fc6dd
update poc, preprocess blocks before inserting to generate missing he…
NazariiDenha Apr 29, 2024
794c251
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
NazariiDenha Apr 29, 2024
5f11287
refactor code to new design with pipeline
NazariiDenha May 17, 2024
885bd3e
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
NazariiDenha May 17, 2024
6b9e837
adapt for codecv0 types
NazariiDenha May 20, 2024
c6201c3
prepare for blob dta asource
NazariiDenha May 21, 2024
27a4e51
implement blob fethcer
NazariiDenha May 24, 2024
6991da0
implement blob fethcer
NazariiDenha May 24, 2024
e483219
fixes after testing and add synced l1 height to db
NazariiDenha Jun 7, 2024
48dd5d8
Merge branch 'feat/sync-directly-from-da' of github.com:scroll-tech/g…
NazariiDenha Jun 7, 2024
eb377fe
fixes and improvements after testing
NazariiDenha Jun 11, 2024
5a48000
fixes and improvements after testing
NazariiDenha Jun 11, 2024
01e5f9c
Merge branch 'feat/sync-directly-from-da' of github.com:scroll-tech/g…
NazariiDenha Jun 11, 2024
1a84506
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
NazariiDenha Jun 11, 2024
fd6d899
update codec version
NazariiDenha Jun 11, 2024
9b929ff
lint
NazariiDenha Jun 11, 2024
9b13959
update codec dependency
NazariiDenha Jun 17, 2024
20ca22d
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
NazariiDenha Jun 17, 2024
bf1d02e
goimports
NazariiDenha Jun 17, 2024
bc9b360
small fixes
NazariiDenha Jun 18, 2024
e55416e
fix lint, turn off handler, worker, miner during da syncing
NazariiDenha Jun 18, 2024
1ff0973
remove comments
NazariiDenha Jul 2, 2024
b3726ff
support blocknative
NazariiDenha Jul 8, 2024
6e0ccdb
fix
NazariiDenha Jul 9, 2024
cc65d00
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
NazariiDenha Jul 9, 2024
49c3c15
fix lint
NazariiDenha Jul 9, 2024
e825a42
fix bug with decoding [parentbatch
NazariiDenha Jul 14, 2024
750c615
support codec v3
NazariiDenha Jul 15, 2024
e3ced13
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
NazariiDenha Jul 24, 2024
cda4058
address comments
NazariiDenha Jul 24, 2024
f3a1c5d
fix: nil pointer when enabling --da.sync=true
jonastheis Jul 24, 2024
2931099
fix: block not being inserted due to Clique verification
jonastheis Jul 25, 2024
1c36c2c
refactor: compose DA types for more cohesion, maintainability and cod…
jonastheis Jul 30, 2024
e587c98
feat: execute blocks only once
jonastheis Jul 26, 2024
9dc17cb
refactor: introduce partial header and partial block for data from DA…
jonastheis Jul 26, 2024
dbdbc88
minor cleanup
jonastheis Jul 26, 2024
26dbf42
feat: fix issue with not specifying difficulty
jonastheis Jul 29, 2024
0f8e35c
refactor: compose DA types for more cohesion, maintainability and cod…
jonastheis Jul 25, 2024
912af16
feat: implement generic min heap
jonastheis Jul 29, 2024
bdb575d
feat: use generic min heap instead of map in BatchQueue
jonastheis Jul 29, 2024
362d160
fix compile errors after rebase
jonastheis Jul 30, 2024
329dd54
disable all p2p networking
NazariiDenha Aug 4, 2024
15ec7c4
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
NazariiDenha Aug 4, 2024
d958069
chore: auto version bump [bot]
NazariiDenha Aug 4, 2024
69e1a9e
v3 finalization
NazariiDenha Aug 4, 2024
0fd2a08
feat: add shrinking map that shrinks itself after a certain number of…
jonastheis Aug 6, 2024
3f68f2c
feat: use shrinking map in batch queue to delete specific elements fr…
jonastheis Aug 6, 2024
53bdb18
feat: execute blocks only once
jonastheis Jul 26, 2024
5243d59
refactor: introduce partial header and partial block for data from DA…
jonastheis Jul 26, 2024
7018b5b
minor cleanup
jonastheis Jul 26, 2024
fe33d83
feat: fix issue with not specifying difficulty
jonastheis Jul 29, 2024
d279e83
Merge remote-tracking branch 'origin/feat/sync-directly-from-da' into…
jonastheis Aug 6, 2024
d3ca47e
feat: implement simple pipeline reset (#941)
jonastheis Aug 6, 2024
19ed11d
feat: remove changes to Clique as we're not verifying signatures when…
jonastheis Aug 6, 2024
a6461d5
feat: only request finalized block number when necessary
jonastheis Aug 6, 2024
85bbc98
minor cleanup and comments
jonastheis Aug 7, 2024
485c229
cycle over list of blob clients (#960)
NazariiDenha Aug 7, 2024
f8eadfe
Merge branch 'develop' into feat/sync-directly-from-da
0xmountaintop Aug 8, 2024
399dcfc
feat: introduce custom errors and mark RPC related errors as temporar…
jonastheis Aug 13, 2024
9e1769f
fix blob_client_list
NazariiDenha Aug 15, 2024
0649954
chore: auto version bump [bot]
NazariiDenha Aug 15, 2024
0b2fe3b
feat: enable prefetching in BuildAndWriteBlock
jonastheis Aug 16, 2024
9e3c838
(follower_node)support beacon node client as blob provider (#988)
NazariiDenha Aug 19, 2024
787c955
update codec version
NazariiDenha Aug 26, 2024
dcd9c5c
Merge branch 'develop' of github.com:scroll-tech/go-ethereum into fea…
NazariiDenha Aug 26, 2024
4a66bf3
support codec v4
NazariiDenha Aug 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions cmd/geth/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,11 @@ var (
utils.CircuitCapacityCheckEnabledFlag,
utils.RollupVerifyEnabledFlag,
utils.ShadowforkPeersFlag,
utils.DASyncEnabledFlag,
utils.DAModeFlag,
utils.DASnapshotFileFlag,
utils.DABlockNativeAPIEndpointFlag,
utils.DABlobScanAPIEndpointFlag,
}

rpcFlags = []cli.Flag{
Expand Down
48 changes: 48 additions & 0 deletions cmd/utils/flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ import (
"github.com/scroll-tech/go-ethereum/p2p/nat"
"github.com/scroll-tech/go-ethereum/p2p/netutil"
"github.com/scroll-tech/go-ethereum/params"
"github.com/scroll-tech/go-ethereum/rollup/da_syncer"
"github.com/scroll-tech/go-ethereum/rollup/tracing"
"github.com/scroll-tech/go-ethereum/rpc"
)
Expand Down Expand Up @@ -864,6 +865,31 @@ var (
Name: "net.shadowforkpeers",
Usage: "peer ids of shadow fork peers",
}

// DA syncing settings
DASyncEnabledFlag = cli.BoolFlag{
Name: "da.sync",
Usage: "Enable node syncing from DA",
}
defaultDA = ethconfig.Defaults.DA.FetcherMode
DAModeFlag = TextMarshalerFlag{
Name: "da.mode",
Usage: `DA sync mode ("l1rpc" or "snapshot")`,
Value: &defaultDA,
}
colinlyguo marked this conversation as resolved.
Show resolved Hide resolved
DASnapshotFileFlag = cli.StringFlag{
Name: "da.snapshot.file",
Usage: "Snapshot file to sync from da",
}
DABlobScanAPIEndpointFlag = cli.StringFlag{
Name: "da.blob.blobscan",
Usage: "BlobScan blob api endpoint",
Value: ethconfig.Defaults.DA.BlobScanAPIEndpoint,
}
DABlockNativeAPIEndpointFlag = cli.StringFlag{
Name: "da.blob.blocknative",
Usage: "BlockNative blob api endpoint",
}
)

// MakeDataDir retrieves the currently requested data directory, terminating
Expand Down Expand Up @@ -1307,6 +1333,9 @@ func SetNodeConfig(ctx *cli.Context, cfg *node.Config) {
setDataDir(ctx, cfg)
setSmartCard(ctx, cfg)
setL1(ctx, cfg)
if ctx.GlobalIsSet(DASyncEnabledFlag.Name) {
cfg.DaSyncingEnabled = ctx.GlobalBool(DASyncEnabledFlag.Name)
}

if ctx.GlobalIsSet(ExternalSignerFlag.Name) {
cfg.ExternalSigner = ctx.GlobalString(ExternalSignerFlag.Name)
Expand Down Expand Up @@ -1581,6 +1610,24 @@ func setEnableRollupVerify(ctx *cli.Context, cfg *ethconfig.Config) {
}
}

func setDA(ctx *cli.Context, cfg *ethconfig.Config) {
if ctx.GlobalIsSet(DASyncEnabledFlag.Name) {
cfg.EnableDASyncing = ctx.GlobalBool(DASyncEnabledFlag.Name)
if ctx.GlobalIsSet(DAModeFlag.Name) {
cfg.DA.FetcherMode = *GlobalTextMarshaler(ctx, DAModeFlag.Name).(*da_syncer.FetcherMode)
}
if ctx.GlobalIsSet(DASnapshotFileFlag.Name) {
cfg.DA.SnapshotFilePath = ctx.GlobalString(DASnapshotFileFlag.Name)
}
if ctx.GlobalIsSet(DABlobScanAPIEndpointFlag.Name) {
cfg.DA.BlobScanAPIEndpoint = ctx.GlobalString(DABlobScanAPIEndpointFlag.Name)
}
if ctx.GlobalIsSet(DABlockNativeAPIEndpointFlag.Name) {
cfg.DA.BlockNativeAPIEndpoint = ctx.GlobalString(DABlockNativeAPIEndpointFlag.Name)
}
}
}

func setMaxBlockRange(ctx *cli.Context, cfg *ethconfig.Config) {
if ctx.GlobalIsSet(MaxBlockRangeFlag.Name) {
cfg.MaxBlockRange = ctx.GlobalInt64(MaxBlockRangeFlag.Name)
Expand Down Expand Up @@ -1656,6 +1703,7 @@ func SetEthConfig(ctx *cli.Context, stack *node.Node, cfg *ethconfig.Config) {
setLes(ctx, cfg)
setCircuitCapacityCheck(ctx, cfg)
setEnableRollupVerify(ctx, cfg)
setDA(ctx, cfg)
setMaxBlockRange(ctx, cfg)
if ctx.GlobalIsSet(ShadowforkPeersFlag.Name) {
cfg.ShadowForkPeerIDs = ctx.GlobalStringSlice(ShadowforkPeersFlag.Name)
Expand Down
51 changes: 51 additions & 0 deletions common/backoff/exponential.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
package backoff

import (
"math"
"math/rand"
"time"
)

// Exponential is a backoff strategy that increases the delay between retries exponentially.
type Exponential struct {
attempt int

maxJitter time.Duration

min time.Duration
max time.Duration
}

func NewExponential(minimum, maximum, maxJitter time.Duration) *Exponential {
return &Exponential{
min: minimum,
max: maximum,
maxJitter: maxJitter,
}
}

func (e *Exponential) NextDuration() time.Duration {
var jitter time.Duration
if e.maxJitter > 0 {
jitter = time.Duration(rand.Int63n(e.maxJitter.Nanoseconds()))
}

minFloat := float64(e.min)
duration := math.Pow(2, float64(e.attempt)) * minFloat

// limit at configured maximum
if duration > float64(e.max) {
duration = float64(e.max)
}

e.attempt++
return time.Duration(duration) + jitter
}

func (e *Exponential) Reset() {
e.attempt = 0
}

func (e *Exponential) Attempt() int {
return e.attempt
}
39 changes: 39 additions & 0 deletions common/backoff/exponential_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
package backoff

import (
"testing"
"time"

"github.com/stretchr/testify/require"
)

func TestExponentialBackoff(t *testing.T) {
t.Run("Multiple attempts", func(t *testing.T) {
e := NewExponential(100*time.Millisecond, 10*time.Second, 0)
expectedDurations := []time.Duration{
100 * time.Millisecond,
200 * time.Millisecond,
400 * time.Millisecond,
800 * time.Millisecond,
1600 * time.Millisecond,
3200 * time.Millisecond,
6400 * time.Millisecond,
10 * time.Second, // capped at max
}
for i, expected := range expectedDurations {
require.Equal(t, expected, e.NextDuration(), "attempt %d", i)
}
})

t.Run("Jitter added", func(t *testing.T) {
e := NewExponential(1*time.Second, 10*time.Second, 1*time.Second)
duration := e.NextDuration()
require.GreaterOrEqual(t, duration, 1*time.Second)
require.Less(t, duration, 2*time.Second)
})

t.Run("Edge case: min > max", func(t *testing.T) {
e := NewExponential(10*time.Second, 5*time.Second, 0)
require.Equal(t, 5*time.Second, e.NextDuration())
})
}
109 changes: 109 additions & 0 deletions common/heap.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
package common

import (
"container/heap"
)

// Heap is a generic min-heap (or max-heap, depending on Comparable behavior) implementation.
type Heap[T Comparable[T]] struct {
heap innerHeap[T]
}

func NewHeap[T Comparable[T]]() *Heap[T] {
return &Heap[T]{
heap: make(innerHeap[T], 0),
}
}

func (h *Heap[T]) Len() int {
return len(h.heap)
}

func (h *Heap[T]) Push(element T) *HeapElement[T] {
heapElement := NewHeapElement(element)
heap.Push(&h.heap, heapElement)

return heapElement
}

func (h *Heap[T]) Pop() *HeapElement[T] {
return heap.Pop(&h.heap).(*HeapElement[T])
}

func (h *Heap[T]) Peek() *HeapElement[T] {
if h.Len() == 0 {
return nil
}

return h.heap[0]
}

func (h *Heap[T]) Remove(element *HeapElement[T]) {
heap.Remove(&h.heap, element.index)
}

func (h *Heap[T]) Clear() {
h.heap = make(innerHeap[T], 0)
}

type innerHeap[T Comparable[T]] []*HeapElement[T]

func (h innerHeap[T]) Len() int {
return len(h)
}

func (h innerHeap[T]) Less(i, j int) bool {
return h[i].Value().CompareTo(h[j].Value()) < 0
}

func (h innerHeap[T]) Swap(i, j int) {
h[i], h[j] = h[j], h[i]
h[i].index, h[j].index = i, j
}

func (h *innerHeap[T]) Push(x interface{}) {
data := x.(*HeapElement[T])
*h = append(*h, data)
data.index = len(*h) - 1
}

func (h *innerHeap[T]) Pop() interface{} {
n := len(*h)
element := (*h)[n-1]
(*h)[n-1] = nil // avoid memory leak
*h = (*h)[:n-1]
element.index = -1

return element
}

// Comparable is an interface for types that can be compared.
type Comparable[T any] interface {
// CompareTo compares x with other.
// To create a min heap, return:
// -1 if x < other
// 0 if x == other
// +1 if x > other
// To create a max heap, return the opposite.
CompareTo(other T) int
}

// HeapElement is a wrapper around the value stored in the heap.
type HeapElement[T Comparable[T]] struct {
value T
index int
}

func NewHeapElement[T Comparable[T]](value T) *HeapElement[T] {
return &HeapElement[T]{
value: value,
}
}

func (h *HeapElement[T]) Value() T {
return h.value
}

func (h *HeapElement[T]) Index() int {
return h.index
}
40 changes: 40 additions & 0 deletions common/heap_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
package common

import (
"testing"

"github.com/stretchr/testify/require"
)

type Int int

func (i Int) CompareTo(other Int) int {
if i < other {
return -1
} else if i > other {
return 1
} else {
return 0
}
}

func TestHeap(t *testing.T) {
h := NewHeap[Int]()

require.Equal(t, 0, h.Len(), "Heap should be empty initially")

h.Push(Int(3))
h.Push(Int(1))
h.Push(Int(2))

require.Equal(t, 3, h.Len(), "Heap should have three elements after pushing")

require.EqualValues(t, 1, h.Pop(), "Pop should return the smallest element")
require.Equal(t, 2, h.Len(), "Heap should have two elements after popping")

require.EqualValues(t, 2, h.Pop(), "Pop should return the next smallest element")
require.Equal(t, 1, h.Len(), "Heap should have one element after popping")

require.EqualValues(t, 3, h.Pop(), "Pop should return the last element")
require.Equal(t, 0, h.Len(), "Heap should be empty after popping all elements")
}
Loading