Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use partition ids with lightweight delete #128

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

iuwqyir
Copy link
Collaborator

@iuwqyir iuwqyir commented Dec 13, 2024

TL;DR

Added _partition_id to DELETE query in ClickHouse connector to optimize data deletion operations.

What changed?

Modified the deleteBatch function to include _partition_id in the DELETE query's WHERE clause. The partition ID is now set to match the chain_id, which helps ClickHouse identify the specific partition for deletion.

How to test?

  1. Execute DELETE operations on ClickHouse tables using the connector
  2. Verify that deletions complete successfully
  3. Check query performance metrics to confirm improved deletion efficiency
  4. Ensure data integrity is maintained across different chain IDs

Why make this change?

ClickHouse performs better when DELETE operations include partition information. By adding the _partition_id to the query, we enable ClickHouse to quickly identify and remove data from specific partitions instead of scanning the entire table, resulting in more efficient delete operations.

@iuwqyir iuwqyir requested a review from AmineAfia December 13, 2024 13:53
Copy link
Collaborator Author

iuwqyir commented Dec 13, 2024

This stack of pull requests is managed by Graphite. Learn more about stacking.

@iuwqyir iuwqyir marked this pull request as ready for review December 13, 2024 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants