Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory consumption for data slicing into objects #2719

Merged
merged 2 commits into from
Jan 30, 2024
Merged

Reduce memory consumption for data slicing into objects #2719

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
08a531a
cli/object: Set payload size field in `put` command
cthulhu-rider Dec 19, 2023
153d56a
node/object: Optimize memory allocs for objects with known payload size
cthulhu-rider Dec 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Changelog for NeoFS Node
- IR does not create new notary requests for the SN's bootstraps but signs the received ones instead (#2717)
- IR can handle third-party notary requests without reliance on receiving the original one (#2715)
- SN validates container session token's issuer to be container's owner (#2466)
- Storage node now consumes much less memory when slicing small data of a fixed size (#2719)

### Removed
- Deprecated `neofs-adm [...] inspect` commands (#2603)
Expand Down
5 changes: 5 additions & 0 deletions cmd/neofs-cli/modules/object/put.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,11 @@ func putObject(cmd *cobra.Command, _ []string) {
cnr, _ = objTemp.ContainerID()
ownerID = *objTemp.OwnerID()
} else {
fi, err := f.Stat()
common.ExitOnErr(cmd, "read file stat: %w", err)

obj.SetPayloadSize(uint64(fi.Size()))

readCID(cmd, &cnr)
ownerID = user.ResolveFromECDSAPublicKey(pk.PublicKey)
}
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ require (
github.com/nspcc-dev/neo-go v0.104.0
github.com/nspcc-dev/neofs-api-go/v2 v2.14.0
github.com/nspcc-dev/neofs-contract v0.19.1
github.com/nspcc-dev/neofs-sdk-go v1.0.0-rc.11.0.20231121125354-d72d76ccb0ba
github.com/nspcc-dev/neofs-sdk-go v1.0.0-rc.11.0.20240130073207-03ed6db7e1cd
github.com/nspcc-dev/tzhash v1.7.1
github.com/olekukonko/tablewriter v0.0.5
github.com/panjf2000/ants/v2 v2.8.2
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,8 +257,8 @@ github.com/nspcc-dev/neofs-contract v0.19.1 h1:U1Uh+MlzfkalO0kRJ2pADZyHrmAOroC6K
github.com/nspcc-dev/neofs-contract v0.19.1/go.mod h1:ZOGouuwuHpgvYkx/LCGufGncIzEUhYEO18LL4cWEbyw=
github.com/nspcc-dev/neofs-crypto v0.4.0 h1:5LlrUAM5O0k1+sH/sktBtrgfWtq1pgpDs09fZo+KYi4=
github.com/nspcc-dev/neofs-crypto v0.4.0/go.mod h1:6XJ8kbXgOfevbI2WMruOtI+qUJXNwSGM/E9eClXxPHs=
github.com/nspcc-dev/neofs-sdk-go v1.0.0-rc.11.0.20231121125354-d72d76ccb0ba h1:dQDBBqmT2rlJGgkeeF3UurB3VgG7buMHDHcz5UQ3bro=
github.com/nspcc-dev/neofs-sdk-go v1.0.0-rc.11.0.20231121125354-d72d76ccb0ba/go.mod h1:2PKUuH7kQaAmQ/USBgmiD/k08ssnSvayor6JAFhrC1c=
github.com/nspcc-dev/neofs-sdk-go v1.0.0-rc.11.0.20240130073207-03ed6db7e1cd h1:kRIn6i7BTa55ae4cH+UcqRfH//XC20mSC4E9WcWxkmM=
github.com/nspcc-dev/neofs-sdk-go v1.0.0-rc.11.0.20240130073207-03ed6db7e1cd/go.mod h1:2PKUuH7kQaAmQ/USBgmiD/k08ssnSvayor6JAFhrC1c=
github.com/nspcc-dev/rfc6979 v0.2.0 h1:3e1WNxrN60/6N0DW7+UYisLeZJyfqZTNOjeV/toYvOE=
github.com/nspcc-dev/rfc6979 v0.2.0/go.mod h1:exhIh1PdpDC5vQmyEsGvc4YDM/lyQp/452QxGq/UEso=
github.com/nspcc-dev/tzhash v1.7.1 h1:6zmexLqdTF/ssbUAh7XJS7RxgKWaw28kdNpE/4UFdEU=
Expand Down
8 changes: 8 additions & 0 deletions pkg/services/object/put/slice.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package putsvc
import (
"context"
"fmt"
"math"

"github.com/nspcc-dev/neofs-node/pkg/services/object/internal"
"github.com/nspcc-dev/neofs-sdk-go/client"
Expand Down Expand Up @@ -60,6 +61,13 @@ func (x *slicingTarget) WriteHeader(hdr *object.Object) error {
opts.CalculateHomomorphicChecksum()
}

if payloadSize := hdr.PayloadSize(); payloadSize != 0 && payloadSize != math.MaxUint64 {
// https://github.com/nspcc-dev/neofs-api/blob/d95228c40283cf6e188073a87a802af7e5dc0a7d/object/types.proto#L93-L95
// zero may be explicitly set and be true, but node previously considered zero
// value as unknown payload, so we keep this behavior for now
opts.SetPayloadSize(payloadSize)
}

var err error
x.payloadWriter, err = slicer.InitPut(x.ctx, &readyObjectWriter{
nextTarget: x.nextTarget,
Expand Down
Loading