Skip to content

Commit

Permalink
Small improvements to fire{chain} tools check forks
Browse files Browse the repository at this point in the history
* `fire{chain} tools check forks` now sorts forks by block number from ascending order (so that line you see is the current highest fork).
* `fire{chain} tools check forks --after-block` can now be used to show only forks after a certain block number.
  • Loading branch information
maoueh committed Jul 28, 2023
1 parent a59daa5 commit b13fa70
Show file tree
Hide file tree
Showing 2 changed files with 38 additions and 12 deletions.
28 changes: 18 additions & 10 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,31 +4,39 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). See [MAINTAINERS.md](./MAINTAINERS.md)
for instructions to keep up to date.

Operators, you should copy/paste content of this content straight to your `firehose-<chain>` project. It is written and meant to be copied over to your project.
Operators, you should copy/paste content of this content straight to your project. It is written and meant to be copied over to your project.

If you were at `firehose-core` version `1.0.0` and are bumping to `1.1.0`, you should copy the content between those 2 version to your own repository.
If you were at `firehose-core` version `1.0.0` and are bumping to `1.1.0`, you should copy the content between those 2 version to your own repository, replacing placeholder value `fire{chain}` with your chain's own binary.

## Next

### Changed

* `fire{chain} tools check forks` now sorts forks by block number from ascending order (so that line you see is the current highest fork).

* `fire{chain} tools check forks --after-block` can now be used to show only forks after a certain block number.

## v0.1.4

This release bumps substreams to v1.1.10
This release bumps Substreams to [v1.1.10](https://github.com/streamingfast/substreams/releases/tag/v1.1.10).

### Fixes
### Fixed

* Fixed: jobs would hang when flags `--substreams-state-bundle-size` and `--substreams-tier1-subrequests-size` had different values. The latter flag has been completely **removed**, subrequests will be bound to the state bundle size.
* Fixed jobs that would hang when flags `--substreams-state-bundle-size` and `--substreams-tier1-subrequests-size` had different values. The latter flag has been completely **removed**, subrequests will be bound to the state bundle size.

### Added

* Added support for *continuous authentication* via the grpc auth plugin (allowing cutoff triggered by the auth system).

## v0.1.3

This release bumps substreams to v1.1.9
This release bumps Substreams to [v1.1.9](https://github.com/streamingfast/substreams/releases/tag/v1.1.9).

### Highlights

#### Substreams Scheduler Improvements for Parallel Processing

The `substreams` scheduler has been improved to reduce the number of required jobs for parallel processing. This affects `backprocessing` (preparing the states of modules up to a "start-block") and `forward processing` (preparing the states and the outputs to speed up streaming in production-mode).
The `substreams` scheduler has been improved to reduce the number of required jobs for parallel processing. This affects `backprocessing` (preparing the states of modules up to a "start-block") and `forward processing` (preparing the states and the outputs to speed up streaming in production-mode).

Jobs on `tier2` workers are now divided in "stages", each stage generating the partial states for all the modules that have the same dependencies. A `substreams` that has a single store won't be affected, but one that has 3 top-level stores, which used to run 3 jobs for every segment now only runs a single job per segment to get all the states ready.

Expand Down Expand Up @@ -70,7 +78,7 @@ The app `substreams-tier1` and `substreams-tier2` should be upgraded concurrentl

#### Operator Changes

* Added `fire<chain> tools check forks <forked-blocks-store-url> [--min-depth=<depth>]` that reads forked blocks you have and prints resolved longest forks you have seen. The command works for any chain, here a sample output:
* Added `fire{chain} tools check forks <forked-blocks-store-url> [--min-depth=<depth>]` that reads forked blocks you have and prints resolved longest forks you have seen. The command works for any chain, here a sample output:

```log
...
Expand All @@ -87,15 +95,15 @@ The app `substreams-tier1` and `substreams-tier2` should be upgraded concurrentl
...
```

* The `fire<chain> tools` commands and sub-commands have better rendering `--help` by hidden not needed global flags with long description.
* The `fire{chain} tools` commands and sub-commands have better rendering `--help` by hidden not needed global flags with long description.

## v0.1.1

#### Operator Changes

* Added missing `--substreams-tier2-request-stats` request debugging flag.

* Added missing Firehose rate limiting options flags, `--firehose-rate-limit-bucket-size` and `--firehose-rate-limit-bucket-fill-rate` to manage concurrent connection attempts to Firehose, check `fire<chain> start --help` for details.
* Added missing Firehose rate limiting options flags, `--firehose-rate-limit-bucket-size` and `--firehose-rate-limit-bucket-fill-rate` to manage concurrent connection attempts to Firehose, check `fire{chain} start --help` for details.

## v0.1.0

Expand Down
22 changes: 20 additions & 2 deletions tools_check.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ import (
"github.com/streamingfast/cli/sflags"
"github.com/streamingfast/dstore"
"github.com/streamingfast/firehose-core/tools"
"golang.org/x/exp/maps"
"golang.org/x/exp/slices"
)

var toolsCheckCmd = &cobra.Command{Use: "check", Short: "Various checks for deployment, data integrity & debugging"}
Expand Down Expand Up @@ -56,6 +58,7 @@ func init() {
toolsCheckMergedBlocksCmd.Flags().BoolP("print-full", "f", false, "Natively decode each block and print the full JSON representation of the block, should be used with a small range only if you don't want to be overwhelmed")

toolsCheckForksCmd.Flags().Uint64("min-depth", 1, "Only show forks that are at least this deep")
toolsCheckForksCmd.Flags().Uint64("after-block", 0, "Only show forks that happened after this block number, if value is not 0")
}

func configureToolsCheckCmd[B Block](chain *Chain[B]) {
Expand Down Expand Up @@ -145,8 +148,22 @@ func toolsCheckForksE(cmd *cobra.Command, args []string) error {
}
}

for _, link := range links {
if len(link) < int(sflags.MustGetUint64(cmd, "min-depth")) {
sortedKeys := maps.Keys(links)
slices.SortFunc(sortedKeys, func(a, b string) bool {
return links[a][0].Num < links[b][0].Num
})

minDepth := sflags.MustGetInt(cmd, "min-depth")
afterBlock := sflags.MustGetUint64(cmd, "after-block")

for _, key := range sortedKeys {
link := links[key]

if len(link) < int(minDepth) {
continue
}

if afterBlock != 0 && link[0].Num <= afterBlock {
continue
}

Expand All @@ -158,6 +175,7 @@ func toolsCheckForksE(cmd *cobra.Command, args []string) error {
if i == 0 {
canonical = " (on chain)"
}

chain[i] = fmt.Sprintf(spaces+"#%d [%s <= %s%s]", blockNumber(segment.Num), segment.ID, segment.PreviousID, canonical)
}

Expand Down

0 comments on commit b13fa70

Please sign in to comment.