Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Event Scrapers API v2 - Save all Txns and Transfer events #147

Merged
merged 67 commits into from
Aug 8, 2024
Merged

Event Scrapers API v2 - Save all Txns and Transfer events #147

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
4ea59fb
Save all transactions.
May 23, 2024
189f558
Adds ERC20 Transfer events. Filters out ERC721 Transfers.
May 25, 2024
7c78d85
Prettier.
May 25, 2024
7bdb072
Filters transactions without calldata.
May 28, 2024
b4fb4ca
Reuses existing Transfer event function signature.
May 29, 2024
792d252
Bugfix: Start scraping Transfer events from 0xSettler deployment block.
May 29, 2024
890169c
Makes 0x EP an optional feature along with all events scraped from it.
May 29, 2024
7530f96
Restores filter for transactions with listened events.
May 29, 2024
ae7981c
Adds support for multiple ABIs for ERC20 Transfer events.
May 29, 2024
94c2c60
Adds env variables for all chains.
May 29, 2024
b13337a
Refactors how optional EP is managed.
May 30, 2024
3b3534d
Changes order of EP validations.
May 30, 2024
ec8ef3b
Optimization. Avoids parsing txns with empty calldata.
May 31, 2024
0b1d812
Adds support for any non-standard ERC20 Transfer event.
Jun 4, 2024
9c02e18
Optimization. Check if calldata is empty before parsing txn.
Jun 4, 2024
69b4164
Merge pull request #148 from 0xProject/andres/make-0xEP-optional
AndresElizondo Jun 4, 2024
0c9a424
Fix some env variables.
Jun 4, 2024
028985b
Adds optional Exchange Proxy flag
Jun 4, 2024
1e7577a
Bugfix. Don't modify original Standard ERC20 Transfer ABI.
Jun 5, 2024
a391759
Cleanup. Handle edge cases.
Jun 5, 2024
816bdce
Improves logging.
Jun 5, 2024
324590a
Adds support for 12 byte ZIDs.
Jun 7, 2024
13dde9e
Adds block filter for 12 byte zid's.
Jun 7, 2024
b614aed
Adds documentation comment.
Jun 7, 2024
84f25bc
Merge pull request #149 from 0xProject/andres/zid-12-bytes
AndresElizondo Jun 7, 2024
d885e49
Save all transactions.
May 23, 2024
8b78acd
Adds ERC20 Transfer events. Filters out ERC721 Transfers.
May 25, 2024
eac4511
Prettier.
May 25, 2024
1ed5647
Filters transactions without calldata.
May 28, 2024
e018a33
Reuses existing Transfer event function signature.
May 29, 2024
9226a2b
Bugfix: Start scraping Transfer events from 0xSettler deployment block.
May 29, 2024
d862ce7
Restores filter for transactions with listened events.
May 29, 2024
321a20c
Adds support for multiple ABIs for ERC20 Transfer events.
May 29, 2024
7f38f63
Adds env variables for all chains.
May 29, 2024
c3e4815
Optimization. Avoids parsing txns with empty calldata.
May 31, 2024
cbbe75a
Adds support for any non-standard ERC20 Transfer event.
Jun 4, 2024
71f3303
Optimization. Check if calldata is empty before parsing txn.
Jun 4, 2024
f1148ed
Makes 0x EP an optional feature along with all events scraped from it.
May 29, 2024
bde7403
Refactors how optional EP is managed.
May 30, 2024
01ec059
Changes order of EP validations.
May 30, 2024
b91542c
Fix some env variables.
Jun 4, 2024
f6361e5
Adds optional Exchange Proxy flag
Jun 4, 2024
606ca2f
Bugfix. Don't modify original Standard ERC20 Transfer ABI.
Jun 5, 2024
13f2dd6
Cleanup. Handle edge cases.
Jun 5, 2024
35c6717
Improves logging.
Jun 5, 2024
8f4b863
Merge branch 'andres/scrapers-save-all-txns' of github.com:0xProject/…
Jun 11, 2024
7658856
Adds support for parsing 0xSettler transactions.
Jun 12, 2024
5c1be42
Adds support for Settler ERC721 Transfer events.
Jun 14, 2024
93a666c
Adds new flags to docker-compose.
Jun 14, 2024
1117215
Initial working implementation of RFQ Event with dynamic loading of s…
Jun 21, 2024
128241c
Refactoring. Moves settler contracts into singleton.
Jun 22, 2024
8ed18d5
Adds auto-update when a Settler ERC721 Transfer event is detected.
Jun 22, 2024
bb212a8
Removes unused block numbers from SettlerContract type.
Jun 22, 2024
cb3ee5e
Fix comments.
Jun 22, 2024
7681d71
Adds isInitialized method to singleton. Prevents crashing.
Jun 22, 2024
460d511
Bugfix.
Jun 22, 2024
0c72bf1
Simplifies simpleton usability.
Jun 22, 2024
700b533
Moves auto-update to postprocess.
Jun 22, 2024
75c9ec6
prettier
Jun 22, 2024
007cdc6
Removes unused logs.
Jun 24, 2024
e7b4abc
Simplifies settler contract singleton.
Jun 24, 2024
27a5f11
Merge pull request #150 from 0xProject/andres/add-rfq-event-listener
AndresElizondo Jun 24, 2024
de3af5f
Adds warning when RPC responds without new blocks.
Jul 10, 2024
0f9e99c
Scrape all wrap/unwrap events belonging to 0x.
Jul 19, 2024
dc2ff3d
Filters ERC20 Transfer Events to only 0x transactions.
Jul 19, 2024
26039ed
Removes zid filters for wraps/unwraps and ERC20 transfers.
Jul 19, 2024
586c903
Revert "Removes zid filters for wraps/unwraps and ERC20 transfers."
Aug 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 42 additions & 4 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@ services:
#KAFKA_AUTH_USER: '${KAFKA_AUTH_USER}'
#KAFKA_AUTH_PASSWORD: '${KAFKA_AUTH_PASSWORD}'
SCRAPER_MODE: 'BLOCKS'
FEAT_ZEROEX_EXCHANGE_PROXY: "true"
EP_DEPLOYMENT_BLOCK: 10247094
SETTLER_DEPLOYMENT_BLOCK: 19582253
MAX_BLOCKS_TO_SEARCH: 1000
MAX_BLOCKS_TO_PULL: 100
MAX_TX_TO_PULL: 1000
Expand Down Expand Up @@ -78,6 +80,9 @@ services:
FEAT_SOCKET_BRIDGE_EVENT: "true"
SOCKET_BRIDGE_CONTRACT_ADDRESS: "0x3a23f943181408eac424116af7b7790c94cb97a5"
SOCKET_BRIDGE_EVENT_START_BLOCK: 18473454
FEAT_ERC20_TRANSFER_ALL: "true"
FEAT_SETTLER_ERC721_TRANSFER_EVENT: "true"
FEAT_SETTLER_RFQ_ORDER_EVENT: "true"

event-pipeline-bsc:
depends_on:
Expand All @@ -91,14 +96,19 @@ services:
CHAIN_ID: '56'
POSTGRES_URI: 'postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres/${POSTGRES_DB}'
SCHEMA: 'events_bsc'
MAX_BLOCKS_REORG: 50
SCRAPER_MODE: 'BLOCKS'
MAX_BLOCKS_REORG: 300
#KAFKA_BROKERS: '${KAFKA_BROKERS}'
#KAFKA_SSL: '${KAFKA_SSL}'
#KAFKA_AUTH_USER: '${KAFKA_AUTH_USER}'
#KAFKA_AUTH_PASSWORD: '${KAFKA_AUTH_PASSWORD}'
FEAT_ZEROEX_EXCHANGE_PROXY: "true"
EP_DEPLOYMENT_BLOCK: 5375047
MAX_BLOCKS_TO_SEARCH: 2000
MAX_BLOCKS_TO_PULL: 5000
SETTLER_DEPLOYMENT_BLOCK: 37570574
MAX_BLOCKS_TO_SEARCH: 500
MAX_BLOCKS_TO_PULL: 90
MAX_TX_TO_PULL: 500
BLOCK_FINALITY_THRESHOLD: 0
SECONDS_BETWEEN_RUNS: 1
RESCRAPE_BLOCKS: 10
FEAT_UNISWAP_V2_VIP_SWAP_EVENT: "true"
Expand All @@ -109,6 +119,8 @@ services:
NFT_FEATURE_START_BLOCK: 15860129
FEAT_WRAP_UNWRAP_NATIVE_EVENT: "true"
WRAP_UNWRAP_NATIVE_CONTRACT_ADDRESS: '0xbb4cdb9cbd36b01bd1cbaebf2de08d9173bc095c'
FEAT_ERC20_TRANSFER_ALL: "true"
FEAT_SETTLER_ERC721_TRANSFER_EVENT: "true"

event-pipeline-polygon:
depends_on:
Expand All @@ -118,13 +130,15 @@ services:
dockerfile: Dockerfile
restart: always
environment:
EVM_RPC_URL: '${EVM_RPC_URL_POLYGON}'
EVM_RPC_URL: '${RPC_URL_POLYGON}'
CHAIN_ID: '137'
POSTGRES_URI: 'postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres/${POSTGRES_DB}'
SCHEMA: 'events_polygon'
SCRAPER_MODE: 'BLOCKS'
ENABLE_PROMETHEUS_METRICS: "true"
FEAT_ZEROEX_EXCHANGE_PROXY: "true"
EP_DEPLOYMENT_BLOCK: 14391480
SETTLER_DEPLOYMENT_BLOCK: 55435171
MAX_BLOCKS_TO_SEARCH: 1000
MAX_BLOCKS_TO_PULL: 251
SECONDS_BETWEEN_RUNS: 2
Expand All @@ -140,6 +154,9 @@ services:
WRAP_UNWRAP_NATIVE_CONTRACT_ADDRESS: '0x0d500b1d8e8ef31e21c99d1db9a6444d3adf1270'
FEAT_TOKENS_FROM_TRANSFERS: "true"
MAX_BLOCKS_REORG: 400
FEAT_ERC20_TRANSFER_ALL: "true"
FEAT_ERC20_TRANSFER_ALL: "true"
FEAT_SETTLER_ERC721_TRANSFER_EVENT: "true"

event-pipeline-avalanche:
depends_on:
Expand All @@ -158,7 +175,9 @@ services:
#KAFKA_AUTH_USER: '${KAFKA_AUTH_USER}'
#KAFKA_AUTH_PASSWORD: '${KAFKA_AUTH_PASSWORD}'
ENABLE_PROMETHEUS_METRICS: "true"
FEAT_ZEROEX_EXCHANGE_PROXY: "true"
EP_DEPLOYMENT_BLOCK: 3601700
SETTLER_DEPLOYMENT_BLOCK: 43807883
MAX_BLOCKS_TO_SEARCH: 5000
MAX_BLOCKS_TO_PULL: 2000
SECONDS_BETWEEN_RUNS: 1
Expand All @@ -172,6 +191,8 @@ services:
NFT_FEATURE_START_BLOCK: 11816835
FEAT_WRAP_UNWRAP_NATIVE_EVENT: "true"
WRAP_UNWRAP_NATIVE_CONTRACT_ADDRESS: '0xb31f66aa3c1e785363f0875a1b74e27b85fd66c7'
FEAT_ERC20_TRANSFER_ALL: "true"
FEAT_SETTLER_ERC721_TRANSFER_EVENT: "true"


event-pipeline-fantom:
Expand All @@ -192,6 +213,7 @@ services:
#KAFKA_AUTH_PASSWORD: '${KAFKA_AUTH_PASSWORD}'
EP_ADDRESS: "0xDEF189DeAEF76E379df891899eb5A00a94cBC250"
ENABLE_PROMETHEUS_METRICS: "true"
FEAT_ZEROEX_EXCHANGE_PROXY: "true"
EP_DEPLOYMENT_BLOCK: 18855765
MAX_BLOCKS_TO_SEARCH: 2000
MAX_BLOCKS_TO_PULL: 1000
Expand Down Expand Up @@ -222,6 +244,7 @@ services:
#KAFKA_AUTH_USER: '${KAFKA_AUTH_USER}'
#KAFKA_AUTH_PASSWORD: '${KAFKA_AUTH_PASSWORD}'
ENABLE_PROMETHEUS_METRICS: "true"
FEAT_ZEROEX_EXCHANGE_PROXY: "true"
EP_DEPLOYMENT_BLOCK: 9350111
MAX_BLOCKS_TO_SEARCH: 5000
MAX_BLOCKS_TO_PULL: 2000
Expand All @@ -246,7 +269,9 @@ services:
#KAFKA_SSL: '${KAFKA_SSL}'
#KAFKA_AUTH_USER: '${KAFKA_AUTH_USER}'
#KAFKA_AUTH_PASSWORD: '${KAFKA_AUTH_PASSWORD}'
FEAT_ZEROEX_EXCHANGE_PROXY: "true"
EP_DEPLOYMENT_BLOCK: 1691335
SETTLER_DEPLOYMENT_BLOCK: 118320052
MAX_BLOCKS_TO_SEARCH: 1000
MAX_BLOCKS_TO_PULL: 1000
SECONDS_BETWEEN_RUNS: 1
Expand All @@ -260,6 +285,8 @@ services:
FEAT_SOCKET_BRIDGE_EVENT: "true"
SOCKET_BRIDGE_CONTRACT_ADDRESS: "0x3a23f943181408eac424116af7b7790c94cb97a5"
SOCKET_BRIDGE_EVENT_START_BLOCK: 110791294
FEAT_ERC20_TRANSFER_ALL: "true"
FEAT_SETTLER_ERC721_TRANSFER_EVENT: "true"

event-pipeline-arbitrum:
depends_on:
Expand All @@ -277,7 +304,9 @@ services:
#KAFKA_SSL: '${KAFKA_SSL}'
#KAFKA_AUTH_USER: '${KAFKA_AUTH_USER}'
#KAFKA_AUTH_PASSWORD: '${KAFKA_AUTH_PASSWORD}'
FEAT_ZEROEX_EXCHANGE_PROXY: "true"
EP_DEPLOYMENT_BLOCK: 4050733
SETTLER_DEPLOYMENT_BLOCK: 197531334
MAX_BLOCKS_TO_SEARCH: 1000
MAX_BLOCKS_TO_PULL: 1000
SECONDS_BETWEEN_RUNS: 30
Expand All @@ -287,6 +316,8 @@ services:
NFT_FEATURE_START_BLOCK: 4050733
FEAT_WRAP_UNWRAP_NATIVE_TRANSFER_EVENT: "true"
WRAP_UNWRAP_NATIVE_CONTRACT_ADDRESS: '0x82af49447d8a07e3bd95bd0d56f35241523fbab1'
FEAT_ERC20_TRANSFER_ALL: "true"
FEAT_SETTLER_ERC721_TRANSFER_EVENT: "true"

event-pipeline-base:
depends_on:
Expand All @@ -300,7 +331,10 @@ services:
CHAIN_ID: '8453'
POSTGRES_URI: 'postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres/${POSTGRES_DB}'
SCHEMA: 'events_base'
SCRAPER_MODE: 'BLOCKS'
FEAT_ZEROEX_EXCHANGE_PROXY: "true"
EP_DEPLOYMENT_BLOCK: 1410394
SETTLER_DEPLOYMENT_BLOCK: 12723120
MAX_BLOCKS_TO_SEARCH: 1000
MAX_BLOCKS_TO_PULL: 100
SECONDS_BETWEEN_RUNS: 1
Expand All @@ -313,6 +347,9 @@ services:
#KAFKA_AUTH_PASSWORD: '${KAFKA_AUTH_PASSWORD}'
FEAT_WRAP_UNWRAP_NATIVE_EVENT: "true"
WRAP_UNWRAP_NATIVE_CONTRACT_ADDRESS: '0x4200000000000000000000000000000000000006'
FEAT_ERC20_TRANSFER_ALL: "true"
FEAT_SETTLER_ERC721_TRANSFER_EVENT: "true"
FEAT_SETTLER_RFQ_ORDER_EVENT: "true"

token-scraper-ethereum:
depends_on:
Expand All @@ -333,6 +370,7 @@ services:
LOG_LEVEL: "info"
FEAT_EXCLUSIVE_TOKENS_FROM_TRANSACTIONS: "true"
TOKENS_FROM_TRANSACTIONS_START_BLOCK: 9193266
FEAT_ZEROEX_EXCHANGE_PROXY: "true"
EP_DEPLOYMENT_BLOCK: 10247094
MAX_BLOCKS_TO_SEARCH: 1000
MAX_BLOCKS_TO_PULL: 1000
Expand Down
52 changes: 52 additions & 0 deletions src/abis.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1426,3 +1426,55 @@ export const STAKING_REWARDS_PAID_ABI = {
name: 'RewardsPaid',
type: 'event',
};

export const STANDARD_ERC20_TRANSFER_ABI = {
anonymous: false,
inputs: [
{
indexed: true,
internalType: 'address',
name: 'from',
type: 'address',
},
{
indexed: true,
internalType: 'address',
name: 'to',
type: 'address',
},
{
indexed: false,
internalType: 'uint256',
name: 'value',
type: 'uint256',
},
],
name: 'Transfer',
type: 'event',
};

export const SETTLER_ERC721_TRANSFER_ABI = {
anonymous: true,
inputs: [
{
indexed: true,
internalType: 'address',
name: 'from',
type: 'address',
},
{
indexed: true,
internalType: 'address',
name: 'to',
type: 'address',
},
{
indexed: true,
internalType: 'uint256',
name: 'tokenId',
type: 'uint256',
},
],
name: 'SettlerERC721Transfer',
type: 'event',
};
Loading
Loading