Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

See if we can avoid returning scan batches that contain no data #234

Open
nicklan opened this issue May 31, 2024 · 0 comments
Open

See if we can avoid returning scan batches that contain no data #234

nicklan opened this issue May 31, 2024 · 0 comments
Labels
good first issue Good for newcomers

Comments

@nicklan
Copy link
Collaborator

nicklan commented May 31, 2024

In debugging #233 , I realized we have a case where the first batch of data does include and Add file, but that it's filtered out due to not matching the predicate that's been pushed down.

We can't just check if all add files are masked, because we just have an EngineData, but perhaps the visitor could remember the indexes of all the add files, and if they are all filtered out, just go on to the next batch. We already do this in log_replay if the entire selection vector is false, but in this case, it wasn't because the other data item was a Remove.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
Development

No branches or pull requests

1 participant