Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve reindex query #2131

Merged
merged 5 commits into from
Oct 31, 2023
Merged

Improve reindex query #2131

merged 5 commits into from
Oct 31, 2023

Conversation

jiqiang90
Copy link
Contributor

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist

  • I have tested locally
  • I have performed a self review of my changes
  • Updated any relevant documentation
  • Linked to any relevant issues
  • I have added tests relevant to my changes
  • Any dependent changes have been merged and published in downstream modules
  • My code is up to date with the base branch
  • I have updated relevant changelogs. We suggest using chan

// const offset = 0;
let completed = false;
// eslint-disable-next-line no-constant-condition
while (!completed) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use sqlIterator there? It seems expensive to load before removing.

This maybe useful. Static destroy returns the number of items destroyed https://sequelize.org/api/v6/class/src/model.js~model#static-method-destroy

I see the same is happening with other data.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sqlIterator apply sql to whole table, it is useful for sql like create index xxx for all records.

In batchDeletePoi, we only need to remove records with certain block ranges. Compare to sqlIteraor, even we add where conditions to excute the block range in sql, it will still iterate the complete table. I think this is the better approach.

packages/node-core/src/indexer/poi/poi.service.ts Outdated Show resolved Hide resolved
packages/node-core/src/indexer/store.service.ts Outdated Show resolved Hide resolved
@stwiname stwiname merged commit ea0277d into main Oct 31, 2023
1 of 2 checks passed
@jiqiang90 jiqiang90 mentioned this pull request Nov 5, 2023
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants