Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vector based on lgalloc for flat container #27880

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

antiguru
Copy link
Member

@antiguru antiguru commented Jun 25, 2024

Introduce containers that allow flat container regions to spill to disk.

Depends on:

Motivation

Tips for reviewer

Checklist

@antiguru antiguru force-pushed the flat_lgalloc branch 3 times, most recently from aa63bb2 to 60c7e2d Compare July 8, 2024 19:26
@antiguru antiguru requested a review from sdht0 July 8, 2024 19:32
@antiguru antiguru force-pushed the flat_lgalloc branch 3 times, most recently from d5a39fb to 08111dd Compare July 9, 2024 17:08
@antiguru antiguru force-pushed the flat_lgalloc branch 3 times, most recently from c249582 to b685d5e Compare July 23, 2024 00:58
antiguru added 8 commits July 23, 2024 14:54
Signed-off-by: Moritz Hoffmann <[email protected]>
Signed-off-by: Moritz Hoffmann <[email protected]>
Signed-off-by: Moritz Hoffmann <[email protected]>
Signed-off-by: Moritz Hoffmann <[email protected]>
@antiguru
Copy link
Member Author

antiguru commented Jul 23, 2024

This PR introduces a small performance regression, which can be explained (in the future prevented.) At the moment, we're using vectors to store on dataflow edges, which have the nice property that they can be preallocated to a specific capacity. This concept doesn't (yet) exist in flat containers. Ideally, we'd like to preallocate each region to its desired size, but without knowing the actual data, we cannot know the size of the regions.

The performance regression only affects the introspection dataflows, and seems to be in the range of 1-2%.

A potential way forward is to introduce a TryPush (c.f. antiguru/flatcontainer#54) trait that guarantees pushing without reallocation, combined with a way to preallocate regions with sufficient capacity, but more work is needed to enable this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant