Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor data insertion logic #89

Conversation

iuwqyir
Copy link
Collaborator

@iuwqyir iuwqyir commented Oct 4, 2024

TL;DR

Refactored data insertion process to allow the storage decide how to handle batched data.
E.g sql databases can use transactions and concurrency whereas memory storage can do everything sequentially

What changed?

  • Introduced a new InsertDataForBlocks method in the IMainStorage interface.
  • Removed individual insert methods for blocks, transactions, logs, and traces from the interface.
  • Implemented InsertDataForBlocks in both ClickHouseConnector and MemoryConnector.
  • Updated the Committer to use the new InsertDataForBlocks method instead of saveDataToMainStorage.
  • Renamed insert methods in ClickHouseConnector and MemoryConnector to be private.

@iuwqyir iuwqyir changed the title change main storage interface to handle inserts all at once Refactor data insertion logic for improved concurrency and error handling Oct 4, 2024
@iuwqyir iuwqyir changed the title Refactor data insertion logic for improved concurrency and error handling Refactor data insertion logic Oct 4, 2024
@iuwqyir iuwqyir requested a review from AmineAfia October 4, 2024 16:38
@iuwqyir iuwqyir marked this pull request as ready for review October 4, 2024 16:39
@iuwqyir iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from d0e0f6f to 4773497 Compare October 4, 2024 16:46
@iuwqyir iuwqyir force-pushed the 10-04-refactor_rpc_to_a_wrapper_for_easier_reuse branch from 5b416ef to 8663df7 Compare October 7, 2024 11:04
@iuwqyir iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch 2 times, most recently from d06c5d1 to d0961cb Compare October 7, 2024 11:44
@iuwqyir iuwqyir requested a review from AmineAfia October 7, 2024 11:47
@iuwqyir iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from d0961cb to b11b079 Compare October 7, 2024 13:19
@iuwqyir iuwqyir force-pushed the 10-04-refactor_rpc_to_a_wrapper_for_easier_reuse branch from 8663df7 to 01aba8e Compare October 7, 2024 20:01
@iuwqyir iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from b11b079 to dd391a7 Compare October 7, 2024 20:01
@iuwqyir iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from dd391a7 to ccac684 Compare October 8, 2024 23:32
@iuwqyir iuwqyir force-pushed the 10-04-refactor_rpc_to_a_wrapper_for_easier_reuse branch from 01aba8e to 49fe94d Compare October 10, 2024 12:48
@iuwqyir iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from ccac684 to 6ead9a7 Compare October 10, 2024 12:48
Copy link
Collaborator Author

iuwqyir commented Oct 10, 2024

Merge activity

  • Oct 10, 10:01 AM EDT: A user started a stack merge that includes this pull request via Graphite.
  • Oct 10, 10:03 AM EDT: Graphite rebased this pull request as part of a merge.
  • Oct 10, 10:04 AM EDT: A user merged this pull request with Graphite.

@iuwqyir iuwqyir changed the base branch from 10-04-refactor_rpc_to_a_wrapper_for_easier_reuse to graphite-base/89 October 10, 2024 14:01
@iuwqyir iuwqyir changed the base branch from graphite-base/89 to main October 10, 2024 14:01
@iuwqyir iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from 6ead9a7 to 0f12a69 Compare October 10, 2024 14:02
@iuwqyir iuwqyir merged commit 5f3c0f7 into main Oct 10, 2024
4 checks passed
@iuwqyir iuwqyir deleted the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch October 10, 2024 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants