Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use new database layout #75

Merged
merged 41 commits into from
Oct 14, 2024
Merged

Use new database layout #75

merged 41 commits into from
Oct 14, 2024

Conversation

fhenneke
Copy link
Contributor

@fhenneke fhenneke commented Oct 9, 2024

This is a work-in-progress PR for using the database tables. It is not meant for review but more as a record of the current state of the implementation.

Current state

  • runs locally, alongside old functionality
  • stores timestamps for transactions
  • stores all tokens which emitted a transfer event
    • does not store ETH yet
    • also stores tokens traded between AMMs
  • stores (at most) one ALL available prices per token
  • stores decimals for all new tokens

Next steps

  • disentangle logic between different data fetching steps
  • add correct handling of errors (data fetching, data writing, ...)

@fhenneke fhenneke changed the base branch from main to add_new_database October 9, 2024 09:35
@fhenneke fhenneke marked this pull request as ready for review October 9, 2024 14:53
@fhenneke fhenneke changed the base branch from add_new_database to main October 9, 2024 14:57
src/helpers/database.py Outdated Show resolved Hide resolved
@@ -71,3 +74,34 @@ def get_auction_id(self, tx_hash: str) -> int:
# convert bytes to int
auction_id = int.from_bytes(call_data_bytes[-8:], byteorder="big")
return auction_id


def get_transaction_timestamp(tx_hash: str, web3: Web3) -> tuple[str, int]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason these are not defined as methods of the BlockchainData class?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be made a method of that class.

I just moved it around a bit and did not want to have it depend on all the other things for computing imbalances or even a chain name in this blockchain fetcher. Testing was a bit easier if those methods are standalone.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fhenneke
Copy link
Contributor Author

The database setup seems to be working more or less now.

Testing price fetching currently seems to require api keys. The coingecko price fetcher cannot be set up since the token list cannot be fetched. (The actual price fetching contains special treatment for not having a key, but that part of the code cannot be reached.)

Copy link

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package New capabilities Transitives Size Publisher
pypi/[email protected] environment, eval, filesystem, network, shell 0 691 kB piro
pypi/[email protected] environment, eval, filesystem, unsafe 0 4.51 MB Legrandin
pypi/[email protected] None 0 0 B
pypi/[email protected] None 0 0 B

🚮 Removed packages: pypi/[email protected], pypi/[email protected], pypi/[email protected], pypi/[email protected]

View full report↗︎

@harisang
Copy link
Contributor

Pushed a commit that comments out the old code except for the raw token imbalances calculation. The writing in the new tables is also based in the token imbalances token list, as this is more efficient (as we do not need prices for tokens with an imbalances of exactly zero).

Merging so as to test for a few hours/days. A folllow-up clean-up PR should follow.

Note: all new db tables have been manually created based on the definitions used in this PR.

@harisang harisang changed the title [WIP] Use new database layout Use new database layout Oct 14, 2024
@harisang harisang merged commit eab8179 into main Oct 14, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants