v0.0.5-beta

[v0.0.5-beta]

First chain-agnostic release. THIS IS BETA SOFTWARE. USE AT YOUR OWN RISK.
WE PROVIDE NO BACKWARDS COMPATIBILITY GUARANTEES FOR THIS RELEASE.

See https://github.com/streamingfast/substreams for usage docs.

• Removed local command. See README.md for instructions on how to run locally now. Build sfeth from source for now.
• Changed the remote command to run.
• Changed run command's --substreams-api-key-envvar flag to ``--substreams-api-token-envvar, and its default value is changed from SUBSTREAMS_API_KEY` to `SUBSTREAMS_API_TOKEN`. See README.md to learn how to obtain such tokens.

Previous Changelog

• 1a22a92 Added CHANGELOG.md
• 378d759 Added LICENSE
• 6f6e46a Added README.md
• c6e537a Added notes.
• b2e180d Added some docs. Renamed remote to run.
• 84b6852 Adding Cargo lock to gitignore
• 961d855 Adding decimals
• 86b760d Adding proto wip
• 408277e Adding sum float 64
• 60b5f59 Adding tests and adding sum_float64
• 6b9e5ab Apply table changes to an entity
• 54b7a68 Attempt at integrating protobuf to json rendering upon output.
• bb77f30 Better flags setting.
• 2b128c7 Better manifest
• b47ad60 Better print
• 91e1cd4 Bump prost to 0.10.1
• b246896 Bumped tidwall/gjson to fix sec issue.
• 39ac0f2 Change hard code value of block
• 856060c Changelog update. First draft of goreleaser.
• c396a0f Changing variable names and adding logging
• 2b2abce Closer closer to something functional. Missing the decoding and displaying the rest is pretty much there.
• ae1b219 Compiles. No more deps on sparkle now.
• 05e7960 Decode faster, support unsupported messages.
• 79583a7 Do the decoding at the right place, and load the proto-path.
• 1fbdfac Don't allow running local when not built through sf-ethereum. Only remote.
• 3f31be3 Don't conditionally add the getters
• f29022a Don't fail when proto not available. Show connecting/connected.
• f19fde7 Draft of substreams-native gRPC endpoint.
• bfc1c94 First draft at importing from PCS and splitting in a proper lib
• e753428 First draft at moving proto defs around, to have a first-class Substreams endpoint.
• a792613 First draft of proto for manifests
• cb97e74 First workable draft of multi-module remote sync'ing.
• 655782f Fix imports
• ac1c28d Fix leak between blocks for input sources.
• d47606c Fix tests. SetBuidlder -> SetOutputStore. We need the data to align for bigint
• 853deea Fixed graph building, and renamed a few stream things into modules.
• 1610439 Fixup RPC calls
• 310e60a Fixup printing of those blocks.
• 38b71c8 Fixup rust tests.
• 4526a22 Fixup the registry, and finding the right func.
• c3b725e Fixups
• 2b242c9 Flush stores at stop block too. Fix compilation issues. Fix logs. Options here and there. Cosmetics in builder.go
• 2f33f70 Go generated code
• 4fd59bb Go mod tidy
• 7c144c3 Gracefully terminate. Better print output. Optimize anypb.Any creation.
• 8f73094 Implementing and adding tests for set max int64, bigint, flaot64 and bigfloat
• 0a5fb9d Implementing sum_big_float and adding tests
• c18fa29 Improved docs. Renamed remote to run.
• fc6b067 Initialize the wasm modules within the Firehose thread.
• 03cef88 LocalConfig and RemoteConfig now extends Config
• 785889e Mega hack!
• 19ac176 Merge
• 64a2135 Merge branch 'develop' into feature/proto_manifest
• 025a55f Merge branch 'develop' of github.com:streamingfast/substreams into develop
• aa8db1e Merge branch 'develop' of github.com:streamingfast/substreams into develop
• fd2ebbb Merge branch 'develop' of github.com:streamingfast/substreams into develop
• 947015b Merge branch 'develop' of github.com:streamingfast/substreams into develop
• 835a1cc Merge branch 'develop' of github.com:streamingfast/substreams into develop
• 5cc72f2 Merge branch 'develop' of github.com:streamingfast/substreams into develop
• fb55150 Merge branch 'develop' of github.com:streamingfast/substreams into develop
• 0165628 Merge branch 'develop' of github.com:streamingfast/substreams into develop
• 09b405c Merge branch 'develop' of github.com:streamingfast/substreams into feature/refactor-state-imports
• 11efad5 Merge pull request #1 from streamingfast/master
• 4925463 Merge pull request #6 from streamingfast/feature/waiter_refactoring
• 67b9fab Merge pull request #7 from streamingfast/refactor-instance
• 44eac00 Merge pull request #8 from streamingfast/graphnode_returnhandler
• 31a8cb4 ModuleSignature WIP
• 8fa5ae4 Move return handler out of there. Will be able to be re-used by the front-end to do the same sort of decoding there.
• 5c747ea Much better on-the-fly decoding
• a377ef7 Par dessus
• 19d3238 Pass in deltas (serialized as JSON for now)
• 48aecaa Proper fallback on ENV variables.
• 81bd08b REVERT! now use block payload instead of unmarshal and re marshall it
• 2951140 Recovered cache baby, somehow, baby tukka.
• 3f00f60 Removing time handling log
• 2a47bb9 Rename a few proto stuff
• f8586cc Rename all those freaking fields. Fix mermaid output. Make that state deltas proto buf serialized instead of json.
• a037fe2 Renames
• 6a9136f Restore println's output. Cleaner output.
• 43b99b4 Revert "Initialize the wasm modules within the Firehose thread."
• db70091 Revert "remove all panic"
• c6d7524 Revert "test commit"
• ba27538 Safeguard against WASM data leaking between blocks
• b36d168 Shared engine
• 521a8e8 Sign native code too.
• d85c2f5 Small refactor. Implement SetIfNotExists. Write as strings in storage for ints and floats. Runtime error when stores do not match imported function calls.
• a96619f Start block int64. Better print.
• 10cbc89 Try to pinpoint the odd behavior
• 5287ec4 UPdate
• 4340f80 Update README.md
• 7674fe3 Update the rust package.
• 5964ae4 Updated README.md
• 0528605 Updated loggers for graph-node and rpc packages
• ce9bbdc Updated proto
• 3c37fbb Use protoreflect for dynamic proto message handling.
• 0a7a75c Use startBlock from the manifest now.
• 013ab56 Using require isEqual with testing suite
• fe9c283 WIP
• 312c326 WIP exchange dataloader
• e40a039 WIP for bigfloat
• 63a8106 WIP transform, print
• 104e7ea WIP: have decode/print.go print with the new output format.
• 4f77f84 Wire in the cursor, step still missing for the remote transforms-based version.
• 54bff23 add a more verbose format and resetting time to now in if statement
• c7fb4cd add context to progress tracker
• db8b90b add default case to input mode and set keys to lowercase
• 11e0c44 add eth_call to rpc to wasm machine, using proto
• 54be505 add gen for substreams
• 41d63da add log for rpc call time taken
• fa6ba49 add logger module
• 570c145 add memory leak test
• 48076b9 add missing file
• 0554f76 add prost-build in build dependencies
• 2a21aff add proto generate files and build scripts
• a4eb426 add protobuf
• 2b4fb7e add remote runtime, update cli, cleanup runtime vars
• a4a0405 add set_if_not_exist as extern function
• e42dc23 add tests for graph methods from manifest file and fix some tests
• 348cc22 add time blocks pps
• b5f493e add time to process x number of blocks per second
• df7c321 add timer for rpc
• 9f00432 added a small tool to print some manifest info
• 7077a9b added graph-node definition parsing
• af02435 added initial block to module and start block is now a program argument
• bd02d03 added no-return-handler param
• ecf7264 added param rpc-cache-store-url
• 50ac81c added secondary-rpc-endpoints params
• d2a95ef added set_min support!
• 9fe1dd1 added stack traces log when recovery from panic
• d08b4a8 added stupid fmt.println
• 7f411f8 added substreams Output "wrapper" around return value
• d25dd6b added support to multi rpc endpoints
• 6e5887b added test for graph
• b53083a added unset to table_change operation
• e5608b7 adding logging for types v1 package
• 5ca3e6a adding more logs and adding statesSaveInterval
• f1cb290 adding proto url for local command
• f6f5e93 adding protobufBlockType missing in the LocalConfig constructor
• 97fbd4b adding test orphan node in the test file
• f295730 all broken wip
• 7791c31 better logging
• c6f764c better start block handling
• af01a46 builder io: walk partials func
• 6bf377d builder: compile-time check for interfaces
• 05bbd05 builder: fix call to WriteState
• 40555a6 builder: fix tests
• 1154df0 builder: fix tests
• 92d9dba builder: if partial file starts at module start block, save as full snapshot
• 25521a5 builder: implement merge logic for all types (int64, bigInt, float64, big.Float)
• 982de94 builder: max tests
• ef1eb93 builder: min tests
• 352a891 builder: partial prep work wip
• b60b87b builder: remove bundler stuff
• 96e64bd builder: remove patate log
• df30de1 builder: sum tests
• 2a44a23 builder: unify receiver names
• 84f4eac bump bstream, add ground work for substreams as transform
• a448af6 bump bstream, firehose with new transform patterns
• c77f822 bump dstore
• 515e830 bump dstore
• ae83b68 bump eth-go
• 2889dd3 bump eth-go, now uses context
• c972832 bump substreams to pick up bugfix for gas uint64 deterministic error
• 8db7cc9 change module initialStartBlock to startBlock
• 9a3f3fe check requested start block before wait loop
• dcfa9c9 cherry pick changes from substreams_transform branch and implement onReturn method
• cbc2063 clear todo
• 94c1c35 cli: add command to get ordered list of stores to run in parallel
• a5f31be cli: add stop-block flag
• af5e1f8 create a new store for each instance
• 315403d database changes test update
• 0d3fd72 database squash test cleanup
• e1e8f43 debug/info -> println
• de8c53c deps: use streamingfast fork of graph library
• 5cde373 ensure substreams transform runner catches EOF as non-error
• ce5cb6e ensure transform creates an (empty) cache to prevent nil pointer
• 9013de9 entity package copied over from sparkle
• 3132c4a extern state methods are no longer taking owner ship of given value
• 914371c extra state import resgister to a func
• 5a74e7c fix ModulesDownTo
• 5e17d6d fix SynchronizeStores
• 8afe606 fix argument count
• 5438097 fix compilation issues
• a78e7e8 fix contiguous file checker bug
• cbd1963 fix corrupted memory issue
• 0855181 fix database merge, update tests
• 24e52dc fix file prefix
• 004cf40 fix generate.sh, generate the dummy RPC proto def
• 6e7901c fix get_last not found return values
• 06eb365 fix get_last not found return values
• e916b5d fix go generate ./...
• f4bd406 fix json decoding of special keys
• 1ed793e fix major memory leak and bump bstream
• 2483911 fix manifest to proto
• 294d1f8 fix manifest_tests
• dc7e415 fix module sort fix logs short name and add better logging
• 32abe53 fix native code entry point
• f5a680f fix pipeline logger module name
• 27f6869 fix pipeline, use proto everywhere
• b5101e8 fix pprof
• bdfbdef fix return handler func signature, minor refactoring and fix some compilation issues
• 4d6c278 fix rpc eth_call to return output data not pointer
• 47fc934 fix runtime using protobuf
• 261492c fix state filenames
• ef5a2e6 fix stores map not populated correctly
• d540ea7 fix test
• dd277ec fix test and optimize start block computation
• 1b3b88d fix test sorry!
• 67cecdd fix tooling logger, tooling command usage
• 69231f1 fix update policy
• 5809a92 fix update policy
• 5d03e5a fix waiter tests
• b606d30 fix warnings on Sprintf
• 52dba7f fix/generate transform proto
• cb0b746 fixed tests
• 1e7c1ea fixed tests agains
• 1235079 fixed tests agains after stepd broke everything!
• 247e6c9 gitignore
• 7e86d8d go mod stuff
• 0c42689 go.sum
• 38bf5a9 graph node return handler wip
• 1c56c27 graph: fix sorting
• ea37811 graph: remove group methods for now, fix tests
• c721a33 handler: fix yet another goroutine variable scope issue
• a8bd20c local run refactoring
• 74c184a make test compile but still failing new logger module back by zap log
• 37911dd manifest graph: integrate and fix tests
• da96e7e manifest graph: new struct using graph lib
• 2470ac6 manifest tests
• 12aff7c manifest: fix graph ancestors method
• a12f60c manifest: fix more tests
• 5c768f2 manifest: fix tests
• 57cd407 manifest: fix yaml tag for ProtoType
• 470d988 manifest: modulegraph ModulesDownTo and StoresDownTo now take a list of modules as parameter
• 2aa9b73 manifest: remove connected requirement for graph
• 38e7c7c manifest: simplify regex
• f14d9b5 manifest: test new yaml
• 59c92c2 manifest: use regex to validate module names
• 97a7085 merge deleted prefixes, prevent the use of keys with prefix 0xFF, and of 1 characters keys.
• 20fee58 module hash as folder instead of file prefix
• 2b29cab module hash now use in state file naming
• fc6b9ab module start block calculation. wip
• 27a8cb6 move back engine and store creation into module creation
• 3f9c811 move code type to modules
• c658bf3 move database change from PCS to substreams proto
• 14570cc move database change substreams proto instead of transform
• 1929b15 move duplicate code from buildNative and buildWasm to Build func
• 0e87839 move substream crate to its own git repo
• a1177d9 now use block payload instead of unmarshal and re marshall it
• 4df49f2 now used string to serialize sum_big_int to heap
• 8644771 only use waiter when in partial mode.
• f958d5e partial mode, waiting for parent stores, refactor of store
• 60c441b partial mode: do not write other stores. save merge result as full image
• 1b6a4a6 partial mode: wait for kv files of parallel processes
• 7d88a32 passing in indirect value of rv in applyTableChange and adding more logging information for clarity
• 7adc8bf pb generate
• 1ad428f pbsubstreams.StateDelta
• 4ebd14b pbsubstreams: add DatabaseChanges squash
• 6b5b39a pipeline: fix goroutine loop bug
• 3a6c50f pipeline: fix goroutine scope bug
• f8cc85c pipeline: handle error to WriteState in handler func
• 841686c pipeline: write full state after waiting for KV
• a53620b prevent failure on start-block == moduleStartBlock
• f74d754 progress tracker
• 0c10555 progress tracker refactorings
• 605b80a prostgres store
• 8d481ae proto::decode(...) doesn't take ownership of the message to decode
• d9bfd21 re added HandlerFunc panic recovery code.
• dc8205b refactor global i
• 382efc4 remote command
• eee6574 remote go mod replace
• 764ae1c remove StoreFactory, FactoryInterface, TestSore
• d561d7b remove all panic
• a0ed73b remove all runtime.OSThreadLock
• 8b91e2e remove comment dead code
• cd532f8 remove commented code
• ecacc02 remove double module instantiation
• 38da580 remove old todo
• 82775e0 remove p flag for partial
• 1b80fda remove println from wasm print
• e1740a2 remove pub mod memory and only using memory crate
• c3f9c2b remove requestedBlockNum from build functions params
• 98da76e remove unused zap-box package.
• 20c1277 remove viper, long live cobra, fix start-block
• 2f00f51 removing call to instance.Close() that call vmInstance.Close(). It may have caused this error ``` double free or corruption (fasttop) SIGABRT: abort PC=0x7fd26003303b m=18 sigcode=18446744073709551610 signal arrived during cgo execution
• 1fd2556 removing unused variables
• d11e35c rename manif manifest
• aef848e rename some proto enum constants
• 8a9682e rename some stuff
• a9ecbd0 rename some variables and method function
• f1c1156 renaming data to types and creating new entity when calling get interface
• a1ccbb8 replace force low state by a pipeline option
• d261afd resolve merge value types todo
• 29986e0 return buffer in proto and forget about it in the output to be able to properly write data once calling extern output function
• 97c19e2 return error if the proto marshall fails on the return handler
• 7a54d74 return pointer when eth_call
• 2efd49d rpc cache cleanup and change logging interval
• 1595170 rpc cache refactoring and improvements.
• 0478217 rpc cache refactoring, move cache to eth-go
• 7ac8b11 rpc cache: always save file to disk
• 82448e0 rpc cache: method cleanup
• 514db39 rpc cache: remove irrevelant tests
• 3cfd5ff rpc: fix unit test to remove file system
• d5e47ef save store state every 10000 blocks
• d7978f0 setting precision to 100
• 9bec0c5 small signature refactoring
• cfe63da small tools to dump state
• ba38d72 some comments for eduard
• 37efd41 start block is not optional anymore. added tests for start block computation. start block has max uint64 value by default
• 6ff345c state io updates
• 64552c0 state: StoreBlock fixes
• c47eaa1 state: add logging
• d20fcb4 state: add squash
• ccf3b35 state: block numbers in file names are zero-padded for readability
• 49a8f7c state: cleanup
• d0a30bc state: fix kv file regex
• d463a3e state: fix waiter tests
• bf78bf4 state: implement all setter interfaces
• c4a08e1 state: implement merge strategies
• 7572492 state: more data stored in special keys on disk. added command to merge partial files in store.
• 1c10fa6 state: more logging
• d4b537d state: store update strategy and value type in special keys
• b81571b state: update ignore merge policy
• dc008c6 state_get_last state_get_first now write to output ptr
• 152f0cf store are now created in build func instead of SynchronizeStore
• eccf277 store: include module start block in full kv state file
• 6bbabf6 strconv ParseFloat bitsize from 100 to 64
• b4b0bd4 substream crate code build tooling to test
• e6bee7b synchronize is now using pipeline stores list
• 070b302 test commit
• 8753d8e testing mono repos
• 32d37f0 testing mono repos take 2. with compilation error
• 252e469 tidy up
• 475d7f3 tools: added merge and cleanup tools. remove old command in cli
• 5b26231 transforms: some more WIP in Run() cmd
• 3184b3a update runtime configurations
• 0cf5039 use switch instead of if statement to check module kind.
• ff1cc7a waiter refactor
• 534f33d waiter: do not wait if target block less than or equal to module start block
• 5ab6457 waiter: fix file prefix in wait func
• 7edd264 wip
• da94ab4 wip
• 2f6adbf wip for alex