Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiepoch #28

Merged
merged 111 commits into from
Nov 5, 2023
Merged
Show file tree
Hide file tree
Changes from 51 commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
6c44b8f
Begin adding multiepoch RPC support
gagliardetto Jul 21, 2023
c52bc3e
Stable tx order inside block
gagliardetto Jul 22, 2023
59d0287
Cleanup
gagliardetto Jul 22, 2023
a48910a
multiepoch: getSignaturesForAddress
gagliardetto Jul 22, 2023
b84303a
Add script to compress gsfa indexes
gagliardetto Jul 22, 2023
1a9107a
Add returnDataNone
gagliardetto Jul 22, 2023
c9b50c7
Update description
gagliardetto Jul 22, 2023
f439cf7
More logs
gagliardetto Jul 22, 2023
76d3504
Improve mode config
gagliardetto Jul 23, 2023
087cd88
Check if URI is currently supported
gagliardetto Jul 24, 2023
1034832
sig-to-epoch
gagliardetto Aug 1, 2023
de7e519
Allow mono-epoch
gagliardetto Aug 1, 2023
f76f36c
Rename filecoin mode config
gagliardetto Aug 1, 2023
24f6560
Fix text
gagliardetto Aug 1, 2023
103f91f
Update logs
gagliardetto Aug 1, 2023
e512781
Fix GSFA multi-epoch
gagliardetto Aug 2, 2023
839bde1
Fix checks
gagliardetto Aug 2, 2023
89b5174
Fix sig-to-epoch index
gagliardetto Aug 2, 2023
df8dd26
Cleanup options
gagliardetto Aug 2, 2023
719526d
compactindex: use eytzinger
gagliardetto Aug 4, 2023
90fa6b5
Update compactindexes version
gagliardetto Aug 4, 2023
66b072e
Prefetch 1 MiB of data from remote files
gagliardetto Aug 4, 2023
6f9d2cd
Cleanup printing
gagliardetto Aug 4, 2023
9f44568
Debug
gagliardetto Aug 4, 2023
32d75b5
http range cache
gagliardetto Aug 6, 2023
5917a7b
Fix tests
gagliardetto Aug 6, 2023
9ee27d3
Cleanup prefetch
gagliardetto Aug 9, 2023
3405167
Cleanup prefetch
gagliardetto Aug 9, 2023
cf9ac3d
Add debug flag
gagliardetto Aug 9, 2023
ca0dbd1
Fix matching
gagliardetto Aug 9, 2023
00a00a9
Fix matching
gagliardetto Aug 9, 2023
69b47f4
Add comments
gagliardetto Aug 9, 2023
dd840c6
Fix file watcher
gagliardetto Aug 9, 2023
d9cb49b
Cleanup
gagliardetto Aug 9, 2023
28a3d01
Cleanup
gagliardetto Aug 9, 2023
2f4b8f2
Add rpc config file watcher that live (re)loads epochs
gagliardetto Aug 10, 2023
2474242
Better handle file events
gagliardetto Aug 10, 2023
2f73eee
Add transparent proxy; closes #29
gagliardetto Aug 10, 2023
194c959
Cleanup logs
gagliardetto Aug 10, 2023
0998d80
Fix loaded addresses field names
gagliardetto Aug 10, 2023
8fab703
Add fallback for HEAD method to get size
gagliardetto Aug 16, 2023
f3c5ddb
Fix log
gagliardetto Aug 16, 2023
0f60ba9
If len is zero, return error
gagliardetto Aug 16, 2023
218db7c
Fix getBlock when parent is in previous epoch
gagliardetto Aug 19, 2023
63cf560
Improve live reload
gagliardetto Aug 19, 2023
c6bca83
Close before replacing or removing
gagliardetto Aug 19, 2023
eadbac4
Improve closing
gagliardetto Aug 19, 2023
7b2dacd
Handle case where parent is in previous epoch
gagliardetto Aug 19, 2023
72cb64d
Fix first block offset (consider header)
gagliardetto Aug 19, 2023
af52d03
Add getVersion enrichment
gagliardetto Aug 19, 2023
a4f0682
Handle getVersion without proxy
gagliardetto Aug 19, 2023
a9995c9
Make sure it's clear when the previousBlockHash
gagliardetto Aug 20, 2023
0816334
Fix token balances and status response
gagliardetto Aug 20, 2023
4564afd
Cleanup printing
gagliardetto Aug 20, 2023
7edfa46
Better ETA
gagliardetto Aug 20, 2023
e032720
Better logs and responses
gagliardetto Aug 20, 2023
c5d944a
Temporarily fix meta parsing
gagliardetto Aug 20, 2023
9952980
Improve debug
gagliardetto Aug 23, 2023
973d3d4
Improve debug
gagliardetto Aug 23, 2023
c05e3e2
Improve debug
gagliardetto Aug 23, 2023
0e1101a
Improve debug
gagliardetto Aug 23, 2023
a24137c
Remove one roundtrip when using remote CAR files
gagliardetto Aug 23, 2023
dc8fb10
Fix url filepath base
gagliardetto Aug 23, 2023
7e42397
Fix tests
gagliardetto Aug 23, 2023
eb0e6f9
If innerInstruction doesn't have index, then set to zero; closes #45
gagliardetto Aug 23, 2023
4f2c07d
Fix preload
gagliardetto Aug 23, 2023
6268deb
Fix rewards values
gagliardetto Aug 23, 2023
1a12294
Add lamports=0 field to rewards
gagliardetto Aug 23, 2023
09be829
More response fixes
gagliardetto Aug 23, 2023
1f132da
More response fixes
gagliardetto Aug 23, 2023
1d5ac57
Move adapters
gagliardetto Aug 23, 2023
889c20b
Better names and comments
gagliardetto Aug 24, 2023
fa56ff4
If sig-to-epoch index is not specified, search all epochs
gagliardetto Aug 30, 2023
0fe316f
Fix concurrent epoch searcher
gagliardetto Sep 1, 2023
6532831
Bucketteer: Initial commit
gagliardetto Sep 5, 2023
03534ca
Refactor bucketteer
gagliardetto Sep 5, 2023
22a7282
Integrate bucketteer into the indexing pipelines
gagliardetto Sep 5, 2023
c5e468e
Cleanup
gagliardetto Sep 5, 2023
ff024fb
Cleanup naming
gagliardetto Sep 5, 2023
d54ff2f
Spawn returns a ok
gagliardetto Sep 5, 2023
3c6000c
Remove sig-to-epoch and use sig-exists instead
gagliardetto Sep 5, 2023
f38c140
Add verification commands
gagliardetto Sep 5, 2023
134023b
Rename
gagliardetto Sep 5, 2023
8aebe93
Add flag to verify index after creating it.
gagliardetto Sep 5, 2023
67f8482
Cleanup names
gagliardetto Sep 5, 2023
07feed9
Use bucketteers in getTransaction
gagliardetto Sep 5, 2023
adbe786
Cleanup debug
gagliardetto Sep 5, 2023
10da770
Fix log
gagliardetto Sep 5, 2023
4056a2a
Improve debug
gagliardetto Sep 5, 2023
577fa33
Warmup sig-exists index
gagliardetto Sep 5, 2023
4a1dbbd
Fix build
gagliardetto Sep 5, 2023
c2bf5a2
Better example
gagliardetto Sep 5, 2023
9d0ec82
Improve timing
gagliardetto Sep 6, 2023
bd5c472
Improve logs
gagliardetto Sep 6, 2023
eef874b
Fix config file watcher
gagliardetto Sep 6, 2023
242b14a
Fix error message
gagliardetto Sep 6, 2023
b14c7f4
Avoid processing the same file twice at the same time
gagliardetto Sep 6, 2023
b260594
Cleanup log
gagliardetto Sep 6, 2023
d24d346
Add support for `json` encoding for transactions
gagliardetto Sep 7, 2023
c8e10f3
Fix `uiTokenAmount` fields
gagliardetto Sep 14, 2023
67ed05d
Fix option parameters
gagliardetto Sep 14, 2023
f2bcc89
Improve dump-car filters
gagliardetto Sep 18, 2023
8d53854
Add concurrency to epoch loading; closes #53
gagliardetto Sep 18, 2023
c24f03f
Add providers field to filecoin config; closes #52
gagliardetto Sep 18, 2023
a03ae27
Add getBlockTime method; closes #50
gagliardetto Sep 18, 2023
3e91762
Proxy failed local requests to remote RPC server; closes #49
gagliardetto Sep 18, 2023
3e63907
Add readahead
gagliardetto Oct 8, 2023
9912cbe
Fix error handling
gagliardetto Oct 8, 2023
3279987
Better error handling
gagliardetto Oct 8, 2023
71ad5b5
Handle the case where the requested amount is larger than the chunk
gagliardetto Oct 8, 2023
f864752
Fix split reader
gagliardetto Oct 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cmd-rpc-server-car-getBlock.go
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ func (t *timer) time(name string) {
t.prev = time.Now()
}

func (ser *rpcServer) handleGetBlock(ctx context.Context, conn *requestContext, req *jsonrpc2.Request) {
func (ser *deprecatedRPCServer) handleGetBlock(ctx context.Context, conn *requestContext, req *jsonrpc2.Request) {
tim := newTimer()
params, err := parseGetBlockRequest(req.Params)
if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion cmd-rpc-server-car-getSignaturesForAddress.go
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ func parseGetSignaturesForAddressParams(raw *json.RawMessage) (*GetSignaturesFor
return out, nil
}

func (ser *rpcServer) handleGetSignaturesForAddress(ctx context.Context, conn *requestContext, req *jsonrpc2.Request) {
func (ser *deprecatedRPCServer) handleGetSignaturesForAddress(ctx context.Context, conn *requestContext, req *jsonrpc2.Request) {
if ser.gsfaReader == nil {
klog.Errorf("gsfaReader is nil")
conn.ReplyWithError(
Expand Down
39 changes: 38 additions & 1 deletion cmd-rpc-server-car-getTransaction.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ func ptrToUint64(v uint64) *uint64 {
return &v
}

func (ser *rpcServer) handleGetTransaction(ctx context.Context, conn *requestContext, req *jsonrpc2.Request) {
func (ser *deprecatedRPCServer) handleGetTransaction(ctx context.Context, conn *requestContext, req *jsonrpc2.Request) {
params, err := parseGetTransactionRequest(req.Params)
if err != nil {
klog.Errorf("failed to parse params: %v", err)
Expand Down Expand Up @@ -159,9 +159,46 @@ func adaptTransactionMetaToExpectedOutput(m map[string]any) map[string]any {
"writable": []any{},
}
}
{
// if has loadedReadonlyAddresses and is []string, then use that for loadedAddresses.readonly
if loadedReadonlyAddresses, ok := meta["loadedReadonlyAddresses"].([]any); ok {
// the address list is base64 encoded; decode and encode to base58
for i, addr := range loadedReadonlyAddresses {
addrStr, ok := addr.(string)
if ok {
decoded, err := base64.StdEncoding.DecodeString(addrStr)
if err == nil {
loadedReadonlyAddresses[i] = base58.Encode(decoded)
}
}
}
meta["loadedAddresses"].(map[string]any)["readonly"] = loadedReadonlyAddresses
delete(meta, "loadedReadonlyAddresses")
}
// if has loadedWritableAddresses and is []string, then use that for loadedAddresses.writable
if loadedWritableAddresses, ok := meta["loadedWritableAddresses"].([]any); ok {
// the address list is base64 encoded; decode and encode to base58
for i, addr := range loadedWritableAddresses {
addrStr, ok := addr.(string)
if ok {
decoded, err := base64.StdEncoding.DecodeString(addrStr)
if err == nil {
loadedWritableAddresses[i] = base58.Encode(decoded)
}
}
}
meta["loadedAddresses"].(map[string]any)["writable"] = loadedWritableAddresses
delete(meta, "loadedWritableAddresses")
}
// remove loadedReadonlyAddresses and loadedWritableAddresses
}
if _, ok := meta["postTokenBalances"]; !ok {
meta["postTokenBalances"] = []any{}
}
if _, ok := meta["returnDataNone"]; !ok {
// TODO: what is this?
meta["returnDataNone"] = nil
}
if _, ok := meta["preTokenBalances"]; !ok {
meta["preTokenBalances"] = []any{}
}
Expand Down
Loading