v2.0.0

• Update to use firehose-core

Upgrading blocks from previous versions:

Run

firenear tools upgrade-merged-blocks <source> <destination> <range>

where <source> is the path to the source directory, <destination> is the path to the destination directory, and <range> is the range of blocks to upgrade.

Notes: the near-firehose-indexer should be updated to releases with core in the release name. This indicates that the indexer is printing the blocks with the new format. This firenear release is compatible with the previous format, but eventually only the new format will be supported.

v1.1.14

• Fixed reader block stats to print properly time of importing block.

v1.1.13

Substreams

• Performance: prevent reprocessing jobs when there is only a mapper in production mode and everything is already cached
• Performance: prevent "UpdateStats" from running too often and stalling other operations when running with a high parallel jobs count
• Performance: fixed bug in scheduler ramp-up function sometimes waiting before raising the number of workers

v1.1.12

• 1.1.12 replaces the deleted release 1.1.11, which should not be used

Operators

Important

We have had reports of older versions of this software creating corrupted merged-blocks-files (with duplicate or extra out-of-bound blocks)
This release adds additional validation of merged-blocks to prevent serving duplicate blocks from the firehose or substreams service.
This may cause service outage if you have produced those blocks or downloaded them from another party who was affected by this bug.

1. Find the affected files by running the following command (can be run multiple times in parallel, over smaller ranges)

tools check merged-blocks-batch <merged-blocks-store> <start> <stop>

2. If you see any affected range, produce fixed merged-blocks files with the following command, on each range:

tools fix-bloated-merged-blocks <merged-blocks-store> <output-store> <start>:<stop>

3. Copy the merged-blocks files created in output-store over to the your merged-blocks-store, replacing the corrupted files.

Added

• Added tools check merged-blocks-batch to simplify checking blocks continuity in batched mode, optionally writing results to a store
• Added the command tools fix-bloated-merged-blocks to try to fix merged-blocks that contain duplicates and blocks outside of their range.
• Command tools print one-block and merged-blocks now supports a new --output-format jsonl format. Bytes data can now printed as hex or base58 string instead of base64 string.
• Added retry loop for merger when walking one block files. Some use-cases where the bundle reader was sending files too fast and the merger was not waiting to accumulate enough files to start bundling merged files
• Firehose logs now include auth information (userID, keyID, realIP) along with blocks + egress bytes sent.

Fixed

• Bumped bstream: the filesource will now refuse to read blocks from a merged-files if they are not ordered or if there are any duplicate.
• The command tools download-from-firehose will now fail if it is being served blocks "out of order", to prevent any corrupted merged-blocks from being created.
• The command tools print merged-blocks did not print the whole merged-blocks file, the arguments were confusing: now it will parse <start_block> as a uint64.
• The command tools unmerge-blocks did not cover the whole given range, now fixed

Removed

• Breaking The reader-node-log-to-zap flag has been removed. This was a source of confusion for operators reporting Firehose on bugs because the node's logs where merged within normal Firehose on logs and it was not super obvious.

  Now, logs from the node will be printed to stdout unformatted exactly like presented by the chain. Filtering of such logs must now be delegated to the node's implementation and how it deals depends on the node's binary. Refer to it to determine how you can tweak the logging verbosity emitted by the node.

• Flag substreams-rpc-endpoints removed, this was present by mistake and unused actually.

• Flag substreams-rpc-cache-store-url removed, this was present by mistake and unused actually.

• Flag substreams-rpc-cache-chunk-size removed, this was present by mistake and unused actually.

v1.1.10

• bump firehose-core to v0.1.12

v1.1.9

Changed

• bump firehose-core to v0.1.10 with regression fix for start block in reversible segment

v1.1.8

Changed

• bump firehose-core to v0.1.10 with new metrics substreams_active_requests and substreams_counter

v1.1.7

• Bumped firehose-core to 0.1.9
• Go version requirement bumped to 1.21

v1.1.6

Important

The Substreams service exposed from this version will send progress messages that cannot be decoded by substreams clients prior to v1.1.12.
Streaming of the actual data will not be affected. Clients will need to be upgraded to properly decode the new progress messages.

Changed

• Bumped firehose-core to 0.1.8
• Bumped substreams to v1.1.12 to support the new progress message format. Progression now relates to stages instead of modules. You can get stage information using the substreams info command starting at version v1.1.12.
• Migrated to firehose-core
• change block reader-node block encoding from hex to base64

Removed

• Removed --substreams-tier1-request-stats and --substreams-tier1-request-stats (substreams request-stats are now always sent to clients)

Fixed

• More tolerant retry/timeouts on filesource (prevent "Context Deadline Exceeded")

v1.1.5-rc1

This release candidate is a hotfix for an issue introduced at v1.1.3 and affecting production-mode where the stream will hang and some map_outputs will not be produced over some specific ranges of the chains.