Reduce memory consumption for data slicing into objects #2719
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
cthulhu-rider
commented
Jan 15, 2024
cthulhu-rider
commented
- based on slicer: Reduce memory costs for payload buffering neofs-sdk-go#539
cthulhu-rider
force-pushed
the
experiment-slicer-optimizaitons
branch
from
074a921 to
fae34e1
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2719 +/- ##
=======================================
Coverage 28.87% 28.87%
=======================================
Files 415 415
Lines 32450 32450
=======================================
Hits 9370 9370
Misses 22237 22237
Partials 843 843 ☔ View full report in Codecov by Sentry. |
cthulhu-rider
requested review from
roman-khimov and
carpawell
as code owners
|
Conflicts and I do not expect it to be merged before nspcc-dev/neofs-sdk-go#539 (this PR is not a draft). |
cthulhu-rider
force-pushed
the
experiment-slicer-optimizaitons
branch
from
fae34e1 to
2e40379
Compare
|
after SDK PR update testcases still pass |
This was referenced Jan 26, 2024
cthulhu-rider
force-pushed
the
experiment-slicer-optimizaitons
branch
from
2e40379 to
71b5df0
Compare
NeoFS API recommends to specify known size of the streamed object so the server could be ready to handle it. NeoFS CLI creates objects with payload from file that have fixed size, so nothing prevents the size from being given. Signed-off-by: Leonard Lyubich <[email protected]>
Previously, storage nodes slicing user data stream into NeoFS objects did not process payload size when it was specified in the request. Since any NeoFS cluster has a system parameter that limits the maximum size of a physically stored object, the node allocated a buffer equal to its value.For "large" objects this was quite acceptable and optimal. However, the smaller the amount of data that was sliced, the more redundant memory was allocated. According to the protocol, the size of the stored data may be unknown, so we cannot always rely on knowing the size of the streaming data. However, in the case when this parameter is specified in the request, nothing prevents the node from using its value to allocate a small enough buffer. In particular, this will significantly reduce memory consumption during mass streaming of “small” data. The updated version of the NeoFS SDK library supports processing a previously known amount of data in-box. All the application needs is to set `SetPayloadSize` slicing option. It's hard to imagine simpler approach. But there were some nuances. According to NeoFS API protocol, a zero header field is considered an explicit setting of the object without a payload, while an unspecified size is passed as all bits set in the field. However, NeoFS nodes for a long time regarded any value, including zero, as a dynamic size. Since zero also means a “forgotten” field, we cannot suddenly begin to perceive it as an object without a payload. Therefore, for update security, only non-zero values other than the maximum possible are processed as the size set by the user. In total, the applied optimization will not cover all cases, but for some of them it significantly increases performance and efficiency. Refs #2686. Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider
force-pushed
the
experiment-slicer-optimizaitons
branch
from
71b5df0 to
153d56a
Compare
cthulhu-rider
changed the title
roman-khimov
approved these changes
Jan 30, 2024
This was referenced Jan 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.