Params per consumer module

[abhimanyusinghgaur]

Right now, one can specify params for a module at a global level in a substreams.yaml file. Like in this manifest, I'm filtering Transfer events using the ethereum_common:filtered_events imported module:

specVersion: v0.1.0
package:
  name: erc20
  version: v0.1.0

imports:
  ethereum_common: https://github.com/streamingfast/substreams-foundational-modules/releases/download/ethereum_common-v0.3.0/ethereum-common-v0.3.0.spkg

protobuf:
  files:
    - erc20/v1/types.proto
  importPaths:
    - proto

binaries:
  default:
    type: wasm/rust-v1
    file: ../target/wasm32-unknown-unknown/release/erc20.wasm

params:
  # filter the Transfer events
  ethereum_common:filtered_events: "evt_sig:0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"

modules:
  - name: map_extract_erc20_transfer_events
    kind: map
    initialBlock: 0
    inputs:
      - map: ethereum_common:filtered_events
    output:
      type: proto:erc20.v1.TransferEvents

  - name: store_unique_erc20_transfer_addresses
    kind: store
    updatePolicy: set_if_not_exists
    valueType: bytes # empty bytes always
    inputs:
      - map: map_extract_erc20_transfer_events

  - name: map_extract_erc20_tokens
    kind: map
    inputs:
      - store: store_unique_erc20_transfer_addresses
        mode: deltas
    output:
      type: proto:erc20.v1.Tokens

  - name: store_erc20_tokens
    kind: store
    updatePolicy: set
    valueType: proto:erc20.v1.TokenInfo
    inputs:
      - map: map_extract_erc20_tokens

Now, I want to build another substream that uses the store_erc20_tokens store and also wants to filter on the PoolCreated event on UniswapV3Factory contract. For that, I will need a manifest like this:

specVersion: v0.1.0
package:
  name: uniswap_v3_pool_tokens
  version: v0.1.0

imports:
  ethereum_common: https://github.com/streamingfast/substreams-foundational-modules/releases/download/ethereum_common-v0.3.0/ethereum-common-v0.3.0.spkg
  erc20: ../erc20/erc20-v0.1.0.spkg

params:
  # filter the PoolCreated events on the UniswapV3factory contract
  ethereum_common:filtered_events: "evt_sig:0x783cca1c0412dd0d695e784568c96da2e9c22ff989357a2e8b1d9b2b4e6b7118 && evt_addr:0x1f98431c8ad98523631ae4a59f267346ea31f984"

modules:
  - name: map_uniswap_v3_pool_tokens
    kind: map
    initialBlock: 12369621
    inputs:
      - map: ethereum_common:filtered_events
      - store: store_erc20_tokens
    output:
      type: proto:someProto

The issue here is that the uniswap_v3_pool_tokens substream overrides the params even for the erc20 substream, resulting in wrong output from the store_erc20_tokens store.

My proposal here is that it should be allowed to specify params at the level of a consumer module so that this use case can work as intended. Eg:

modules:
  - name: map_uniswap_v3_pool_tokens
    kind: map
    initialBlock: 12369621
    inputs:
      - map: ethereum_common:filtered_events
      - store: store_erc20_tokens
    output:
      type: proto:someProto
    params:
      ethereum_common:filtered_events: "evt_sig:0x783cca1c0412dd0d695e784568c96da2e9c22ff989357a2e8b1d9b2b4e6b7118 && evt_addr:0x1f98431c8ad98523631ae4a59f267346ea31f984"

This way, a set of independent params can be applied to a source module when it's being used by multiple consumer modules.

[abhimanyusinghgaur]

Hi @abourget, the absence of this feature seriously limits the reuse of the substreams cache. Because of this, I always end up using ethereum_common:all_events and filter things manually every time instead of just using the ethereum_common:filtered_events with a param. If it was possible to use the ethereum_common:filtered_events multiple time with different params in the same substream, then I would be able to reuse a lot of substreams cache instead of running computations everytime from scratch.

Are there any plans to solve it?