Dune TVP

How to run the code for the first time

Prerequisite

Follow the instructions here to install docker.

Run the following commands:

docker build . -t tvp

Note: You might need to bump RAM settings as the default value of 2GB might not be enough.

• The CLI command all, will make sure to fetch all the data, aggregate it, load it to Dune tables and also give insights on Top analysis.

  docker run --mount type=bind,source="$(pwd)"/data,target=/app/data tvp all <api-key>
  

Interact with the application

Build and Run Docker Container

To build the Docker container:

docker build . -t tvp

For simplicity, there is a CLI to run the application code.

Use the command below to start the container in interactive mode. This is ideal for running multiple CLI commands, and it ensures that any files created within the container are saved to your local data directory.

docker run -it --entrypoint bash --mount type=bind,source="$(pwd)"/data,target=/app/data tvp

Available commands:

python tvp/cli.py --help

• all: Runs all the commands.

  python tvp/cli.py all <api-key>
  

• extract: Extracts raw data from the Dune API for a specific query and time range. By default, runs for a previous day.

  python tvp/cli.py extract <api-key> <query-id>
  

• backfill: Backfills data using Dune API of a specific query execution.

  python tvp/cli.py backfill <api-key> <query-execution-id>
  

• aggregate: Aggregates raw TVP data based on the provided key and writes it to a .parquet file.

  python tvp/cli.py aggregate <key>
  

• top: Displays the top entries based on the specified aggregation key and value. It also loads the results to a Dune table.

  python tvp/cli.py top <key> <aggregated-value>
  

• load: Loads aggregation data to a table using Dune API.

  python tvp/cli.py load <api_key> <key>
  

Run tests

pytest

Parquet File Tools

Use these commands to inspect Parquet files:

• Show the first 10 rows of the Parquet file

parq <file-path> --head 10

• Show the row count of the Parquet file

parq <file-path> -c