Conflux ETL

Conflux ETL is a tool for extracting data from the Conflux blockchain and loading it into a datastore.

It's a fork of the ethereum-etl project.

Supported instructions:

• export_blocks_and_transactions
• export_token_transfers
• export_receipts_and_logs
• export_contracts
• export_tokens
• export_logs
• extract_csv_column
• get_block_range_for_date
• get_keccak_hash

Supported data outputs:

• Console
• CSV files

Welcome to contribute to the project by submitting a pull request.

---

Full documentation available here.

Quickstart

Install Ethereum ETL:

pip3 install .

Export blocks and transactions (Schema, Reference):

> ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://conflux-core-public.unifra.io

Export ERC20 and ERC721 transfers (Schema, Reference):

> ethereumetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri https://conflux-core-public.unifra.io --output token_transfers.csv

---

Stream blocks, transactions, logs, token_transfers continually to console (Reference):

> pip3 install ethereum-etl[streaming]
> ethereumetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri https://conflux-core-public.unifra.io

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e .
> python3 ethereumetl.py

Useful Links

• Schema
• Command Reference
• Documentation
• Public Datasets in BigQuery
• Exporting the Blockchain
• Querying in Amazon Athena
• Querying in Google BigQuery
• Querying in Kaggle
• Airflow DAGs
• Postgres ETL
• Ethereum 2.0 ETL

Running Tests

> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> export PROVIDER_URL=<your_porvider_uri>
> pytest -vv

Running Tox Tests

> pip3 install tox
> tox

Running in Docker

1. Install Docker: https://docs.docker.com/get-docker/

2. Build a docker image > docker build -t ethereum-etl:latest . > docker image ls

3. Run a container out of the image

   > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://conflux-core-public.unifra.io
   > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://conflux-core-public.unifra.io
   

4. Run streaming to console or Pub/Sub

   > docker build -t ethereum-etl:latest .
   > echo "Stream to console"
   > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt
   > echo "Stream to Pub/Sub"
   > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/crypto_ethereum
   

If running on Apple M1 chip add the --platform linux/x86_64 option to the build and run commands e.g.:

docker build --platform linux/x86_64 -t ethereum-etl:latest .
docker run --platform linux/x86_64 ethereum-etl:latest stream --start-block 500000

Projects using Ethereum ETL

• Google - Public BigQuery Ethereum datasets
• Nansen - Analytics platform for Ethereum