Try autoupdating the network

[roman-khimov]

With #2642 solved we can try autoupdate functionality of the network:

• run an old (0.38.0) network with dev-env (contracts 0.17.0)
• create some containers, push some objects into it
• update IR binaries to the current master builds
• check contracts that should be auto-updated (neofs-adm morph dump-hashes)

[evgeniiz321]

@roman-khimov updating of the running env is a little bit tricky. Need some help with these 3 questions.

1. Currently in a running env we also have a morph-chain. Besides updating IR binaries we also need to update IR config and storage nodes config (since they use morph-chain). What is the expected update procedure? Do I really need to update configs, do some other things (like removing morph-chain), or it should work just with a new binary?
2. Storage nodes can't be recreated currently as is (perhaps I'm missing the right way to restart them, but I can't create them from scratch since I have containers with objects there)- I see these errors:

2023/11/20 20:46:49 can not create (ensure it exists) dir by '/storage/peapod0.db' path: mkdir /storage/peapod0.db: not a directory
2023/11/20 20:47:06 can not create (ensure it exists) dir by '/storage/peapod0.db' path: mkdir /storage/peapod0.db: not a directory
2023/11/20 20:47:23 can not create (ensure it exists) dir by '/storage/peapod0.db' path: mkdir /storage/peapod0.db: not a directory
2023/11/20 20:47:35 can not create (ensure it exists) dir by '/storage/peapod0.db' path: mkdir /storage/peapod0.db: not a directory
2023/11/20 20:47:54 can not create (ensure it exists) dir by '/storage/peapod0.db' path: mkdir /storage/peapod0.db: not a directory
2023/11/20 20:48:07 can not create (ensure it exists) dir by '/storage/peapod0.db' path: mkdir /storage/peapod0.db: not a directory
2023/11/20 20:48:21 can not create (ensure it exists) dir by '/storage/peapod0.db' path: mkdir /storage/peapod0.db: not a directory
2023/11/20 20:48:35 can not create (ensure it exists) dir by '/storage/peapod0.db' path: mkdir /storage/peapod0.db: not a directory

What is the proper way of updating storage nodes ? or is this a bug?

3. In tests I'm getting this error after IR update (without storage node config updates, so they keep using old morph-chain):

E           RuntimeError: Failed to init s3 credentials because of error
E           Could not find AWS credentials in output:
E           Enter password for /home/ezayats/neofs-testcases/TemporaryDir/18e74049-c35a-470f-aec9-0d0b070f0b63.json >
E           2023-11-20T20:41:57.564Z	DEBUG	s3-authmate/main.go:703	prepare connection pool
E           2023-11-20T20:41:57.578Z	INFO	authmate/authmate.go:235	check container or create	{"cid": "11111111111111111111111111111111", "friendly_name": "b17c489e-2120-46ba-bdc6-b3f215594d3a", "placement_policy": "REP 2 IN X CBF 3 SELECT 2 FROM * AS X"}
E           failed to issue secret: check container: create container in NeoFS: save container via connection pool: put: status: code = 1024 message = chain/client: contract execution finished with state FAULT; exception: at instruction 2034 (THROW): unhandled exception: "insufficient balance to create container"

Also not sure why the balance went away somewhere, is it a bug, or do I need to do some refills here?

[roman-khimov]

1. morph-chain is not affected. It'll be internalized eventually like in Internal consensus mode of Inner Ring with Sidechain auto-deploy neofs-dev-env#282, but for this test it can work as is, not requiring any changes.
2. Looks like 0.38.0 bug which was fixed in 0.38.1. So you need to start from 0.38.1.
3. That's strange, if you have something before the upgrade it can't evaporate. Maybe you've effectively restarted the chain.

[evgeniiz321]

@roman-khimov here is the detailed version of my upgrade test:

1. Run dev-env (0.38.1 versions) :

631fb5914fe9   nspccdev/neofs-storage:0.38.1                      "neofs-node --config…"   About a minute ago   Up About a minute (healthy)                                        s01
3c02acf6fc0c   nspccdev/neofs-ir:0.38.1                           "neofs-ir --config /…"   About a minute ago   Up About a minute (healthy)                                        ir01

2. Create container, push object - all via s3 gate:

2023-11-22T00:33:52.032Z	info	api/router.go:162	call method	{"status": 200, "host": "s3.neofs.devenv:8080", "request_id": "5a6a0d5e-7e30-42cc-93c0-a9ec43c08976", "method": "GetObject", "bucket": "e9801240-7150-4fc6-b435-99cf76f21d26", "object": "5f4ade10-44d9-40b1-ad95-2e9ab2a4c0d2", "description": "OK"}

3. Update IR config (same as here Internal consensus mode of Inner Ring with Sidechain auto-deploy neofs-dev-env#282), restart IR with a new image
4. IR restarted successfully. Contract versions are new (neofs-adm.yml is updated here as well to use IR instead of old morph-chain, old morph-chain shows old contracts):

neofs-adm targeting IR:
(venv.local-pytest) ezayats@testsrv1:~/neofs-dev-env$ ./vendor/neofs-adm morph dump-hashes --config neofs-adm.yml
nns         (v0.18.0):  b0676d489bb45660af67dd1a546265a5b9679a2e  unknown
alphabet 0  (v0.18.0):  fda9a091007c2a5c08a8f86d5b1cc86960095763  2033-11-19 00:39:10.435 +0000 UTC
audit       (v0.18.0):  35b57f80c4ec6c9c0d849cdf0bc38dc925f312af  2033-11-19 00:38:52.404 +0000 UTC
balance     (v0.18.0):  177731a0b5e794ca34248d686e1662b6b7256b89  2033-11-19 00:38:55.41 +0000 UTC
container   (v0.18.0):  f456f52637795df1ef3164618194da9e14adfa88  2033-11-19 00:39:04.426 +0000 UTC
neofsid     (v0.18.0):  bf6f777ebafa97f62c899c442c1f5f1460b0a35f  2033-11-19 00:39:01.421 +0000 UTC
netmap      (v0.18.0):  58a712b727ce44a5451d102844af2edd3359d8d9  2033-11-19 00:39:07.43 +0000 UTC
proxy       (v0.18.0):  f3fa94808224173d9d36248d142b5d4896417974  2033-11-19 00:38:47.395 +0000 UTC
reputation  (v0.18.0):  90d5f4e4c40c747d5a56af0e293b134b7c61718c  2033-11-19 00:38:58.415 +0000 UTC

neofs-adm targeting morph-chain:
ezayats@testsrv1:~/neofs-dev-env$ ./vendor/neofs-adm morph dump-hashes --config neofs-adm.yml
nns         (v0.17.0):  79d697065a80cb332488c9db1d9b5ba38d9ba0bc  unknown
alphabet 0  (v0.17.0):  34269ea503e495752d62915ca0f154ba7aef92a7  2033-11-19 00:29:34.24 +0000 UTC
audit       (v0.17.0):  3dd128f6ffe307c9fcd2df21f670d39025171234  2033-11-19 00:29:34.24 +0000 UTC
balance     (v0.17.0):  95b7f52b0adde83a0955505d286421a77a168414  2033-11-19 00:29:34.24 +0000 UTC
container   (v0.17.0):  72576c968205e75eabc87708394839f1cf09cd57  2033-11-19 00:29:34.24 +0000 UTC
neofsid     (v0.17.0):  b0aaaac052cc5860b1b79815daae87b3d8d4b849  2033-11-19 00:29:34.24 +0000 UTC
netmap      (v0.17.0):  707516630852f4179af43366917a36b9a78b93a5  2033-11-19 00:29:34.24 +0000 UTC
proxy       (v0.17.0):  f59eb74d61b1df8e1065fe011758a8c55414386d  2033-11-19 00:29:34.24 +0000 UTC
reputation  (v0.17.0):  4b3a8e1e8044affe893855c127fec3880cd5ce76  2033-11-19 00:29:34.24 +0000 UTC

5. Try to read old info from s3 gate, no success:

E           2023-11-22T00:41:12.482Z	DEBUG	s3-authmate/main.go:703	prepare connection pool
E           2023-11-22T00:41:12.502Z	INFO	authmate/authmate.go:235	check container or create	{"cid": "11111111111111111111111111111111", "friendly_name": "ddcd6b9b-4adf-48fb-8255-0e6a806ab9a0", "placement_policy": "REP 2 IN X CBF 3 SELECT 2 FROM * AS X"}
E           failed to issue secret: check container: create container in NeoFS: save container via connection pool: put: status: code = 1024 message = chain/client: contract execution finished with state FAULT; exception: at instruction 2034 (THROW): unhandled exception: "insufficient balance to create container"

6. One extra thing, I'm not sure how to properly update running storage nodes config and whether or not they should be updated in this test, but if just restart them after config update - it results in errors I mentioned above (even on 0.38.1 version).

Here are logs from containers after IR upgrade:
logs.tar.gz

Not sure what else should I do here, and is it an expected result or not. But current upgrade steps don't work as is.

[roman-khimov]

This just means that you've got a new chain and this is exactly what should be avoided. You need to keep IR configuration with separated CNs.

[evgeniiz321]

@roman-khimov without consensus parameters in an IR config there is no autoupdate (https://github.com/nspcc-dev/neofs-node/blob/master/pkg/innerring/innerring.go#L362), so contracts stay the same. When consensus parameters are specified - we get 2 chains

[cthulhu-rider]

contracts stay the same

@evgeniiz321 pls describe how do you check this?

[evgeniiz321]

With this commit c134063, how the IR config should look like during the update procedure?

Currently it asks me to provide 'consensus' block no matter what, otherwise:

invalid blockchain configuration: missing root section 'morph.consensus'

With consensus block specified in config, i got this:

init internal morph client: scope setup: InferHash: Unknown contract (-102)

And an IR container is not starting.

As a base I'm still using this cfg - https://github.com/nspcc-dev/neofs-dev-env/pull/282/files#diff-eef24f7a930f3188178bdd2772c784f8c5e1caa7453bc8d02856dabb18d8b4d5

You can check docker containers here - testsrv1.spb1.nspcc.ru, I left config as is (/home/ezayats/neofs-dev-env)

[cthulhu-rider]

@evgeniiz321

u must configure either morph.endpoints (remote) or morph.consensus (local consensus) section. 1st has higher priority

then, regardless of the consensus, network deployment is toggled via

neofs-node/config/example/ir.yaml

Lines 90 to 92 in c134063

	fschain_autodeploy: true # Optional flag to run auto-deployment procedure of the FS chain. By default,
	# the chain is expected to be deployed/updated in the background (e.g. via NeoFS ADM tool).
	# If set, must be 'true' or 'false'.

init internal morph client: scope setup: InferHash: Unknown contract (-102)

seems like u wanna auto-deployment but fschain_autodeploy flag is missing

[roman-khimov]

@cthulhu-rider: but if it's false, it's supposed to operate normally with the contracts currently on the network, right? IIUC it doesn't start at all right now.

[cthulhu-rider]

operate normally with the contracts currently on the network, right?

yep, but then all contracts must be deployed in advance (which hasn't been done according to Unknown contract error)

@evgeniiz321 pls share used config

[evgeniiz321]

# Logger section
logger:
  level: debug  # Minimum enabled logging level

# Wallet settings
wallet:
  path: /wallet.json  # Path to NEP-6 NEO wallet file
  address: XXX  # Account address in the wallet; ignore to use default address
  password: XXX  # Account password in the wallet

# Profiler section
pprof:
  enabled: true
  address: :port  # Endpoint for application pprof profiling; disabled by default
  shutdown_timeout: 30s  # Timeout for profiling HTTP server graceful shutdown

# Application metrics section
prometheus:
  enabled: true
  address: :port  # Endpoint for application prometheus metrics; disabled by default
  shutdown_timeout: 30s  # Timeout for metrics HTTP server graceful shutdown

# Toggling the sidechain-only mode
without_mainnet: false

# Neo main chain RPC settings
mainnet:
  endpoints: # List of websocket RPC endpoints in mainchain; ignore if mainchain is disabled
    - ws://addr:port/ws

# Neo side chain RPC settings
morph:
  consensus:
    magic: 15405
    committee:
      - XXXX
    storage:
      type: boltdb
      path: ./db/morph.bolt
    time_per_block: 1s
    max_traceable_blocks: 200000
    seed_nodes:
      - :port
    rpc:
      listen:
        - ":port"
    p2p:
      dial_timeout: 3s
      proto_tick_interval:
      listen:
        - ":port"
      peers:
        min: 0
        max: 10
        attempts: 5
      ping:
        interval: 30s
        timeout: 90s
    set_roles_in_genesis: true
  validators:  # List of hex-encoded 33-byte public keys of sidechain validators to vote for at application startup
    - XXXX

network_settings: # NeoFS network settings managed in the Netmap contract
  epoch_duration: 240 # Time interval (approximate) between two adjacent NeoFS epochs measured in Sidechain blocks.
  # Must be an integer in range [1, 18446744073709551615]
  max_object_size: 67108864 # [bytes] Maximum size of physically stored NeoFS objects. Note that this applies
  # only to objects located on storage nodes: user objects have no restrictions and, if necessary, are sliced.
  # Must be an integer in range [1, 18446744073709551615]
  require_homomorphic_hashing: true # Toggles the requirement for homomorphic hashing of object payloads.
  # Must be 'true' or 'false'
  allow_maintenance_mode: true # Toggles permission to transition storage nodes to maintenance state.
  # Must be 'true' or 'false'
  eigen_trust:
    alpha: 0.1 # Alpha parameter of EigenTrust algorithm used in the Reputation system.
    # Must be a floating point number in range [0, 1].
    iterations_number: 4 # Number of EigenTrust algorithm iterations to pass in the Reputation system.
    # Must be an integer in range [1, 18446744073709551615]
  price: # Price settings. NEOFS means NeoFS Balance contract tokens (usually GASe-12).
    storage: 100000000 # [NEOFS] Price for 1GB of data paid every epoch by data owner to storage nodes.
    # Must be an integer in range [0, 18446744073709551615]
    fee:
      ir_candidate: 100 # [GASe-8] Contribution from the new candidate to the Inner Ring. Must be non-negative integer
      # Must be an integer in range [0, 18446744073709551615]
      withdraw: 100000000 # [GASe-8] Fee paid by the user account to;
        # - NeoFS Processing contract (if Notary service is enabled in the NeoFS Mainchain)
      # - each Alphabet member (otherwise)
      # Must be an integer in range [0, 18446744073709551615]
      audit: 10000 # [NEOFS] Fee for data audit paid by storage group owner to the auditor (Inner Ring member).
      # Must be an integer in range [0, 18446744073709551615]
      new_container: 1000 # [NEOFS] Fee for new container paid by creator to each Alphabet member.
      # Must be an integer in range [0, 18446744073709551615]
      container_domain: 500 # [NEOFS] Fee for container's NNS domain paid by container creator to each Alphabet member.
      # Must be a non-negative integer
  custom: # Optional list of custom key-value pairs to be set in the network configuration. Forbidden keys:
    # [AuditFee, BasicIncomeRate, ContainerAliasFee, ContainerFee, EigenTrustAlpha, EigenTrustIterations, EpochDuration,
    # HomomorphicHashingDisabled, InnerRingCandidateFee, MaintenanceModeAllowed, MaxObjectSize, WithdrawFee]
    # Note that this list can be extended in the future, so, to avoid potential collision, it is recommended
    # to use the most specific keys.
    - my_custom_key1=val1
    - my_custom_key2=val2

nns:
  system_email: [email protected]

# Network time settings
timers:
  emit: 50  # Number of sidechain blocks between GAS emission cycles; disabled by default
  stop_estimation:
    mul: 1  # Multiplier in x/y relation of when to stop basic income estimation within the epoch
    div: 4  # Divider in x/y relation of when to stop basic income estimation within the epoch
  collect_basic_income:
    mul: 1  # Multiplier in x/y relation of when to start basic income asset collection within the epoch
    div: 2  # Divider in x/y relation of when to start basic income asset collecting within the epoch
  distribute_basic_income:
    mul: 3  # Multiplier in x/y relation of when to start basic income asset distribution within the epoch
    div: 4  # Divider in x/y relation of when to start basic income asset distribution within the epoch

# Storage node GAS emission settings
emit:
  storage:
    amount: 1000000000  # Fixed8 value of sidechain GAS emitted to all storage nodes once per GAS emission cycle; disabled by default

# Storage node removal settings
netmap_cleaner:
  enabled: true  # Enable voting for removing stale storage nodes from network map
  threshold: 3  # Number of NeoFS epoch without bootstrap request from storage node before it considered stale

# Audit settings
audit:
  pdp:
    max_sleep_interval: 100ms  # Maximum timeout between object.RangeHash requests to the storage node

# Settlement settings
settlement:
  basic_income_rate: 100000000  # Optional: override basic income rate value from network config; applied only in debug mode
  audit_fee: 100000  # Optional: override audit fee value from network config; applied only in debug mode

# LOCODE database
locode:
  db:
    path: /locode/db  # Path to UN/LOCODE database file

[cthulhu-rider]

@evgeniiz321 i dont see fschain_autodeploy flag. That means Inner Ring expects all contracts to be deployed before the app startup moment

do u preset some file by ./db/morph.bolt path or starting from scratch?

[evgeniiz321]

I managed to successfully restart IR with fschain_autodeploy parameter (from scratch, used dev-env master, then updated IR cfg + image version), see updated config:

# Logger section
logger:
  level: debug  # Minimum enabled logging level

# Wallet settings
wallet:
  path: /wallet.json  # Path to NEP-6 NEO wallet file
  address: XXX  # Account address in the wallet; ignore to use default address
  password: XXX  # Account password in the wallet

# Profiler section
pprof:
  enabled: true
  address: :port  # Endpoint for application pprof profiling; disabled by default
  shutdown_timeout: 30s  # Timeout for profiling HTTP server graceful shutdown

# Application metrics section
prometheus:
  enabled: true
  address: :port  # Endpoint for application prometheus metrics; disabled by default
  shutdown_timeout: 30s  # Timeout for metrics HTTP server graceful shutdown

# Toggling the sidechain-only mode
without_mainnet: false

# Neo main chain RPC settings
mainnet:
  endpoints: # List of websocket RPC endpoints in mainchain; ignore if mainchain is disabled
    - ws://XX:port/ws

# Neo side chain RPC settings
morph:
  endpoints: # List of websocket RPC endpoints in sidechain
    - ws://XX:port/ws
  validators:  # List of hex-encoded 33-byte public keys of sidechain validators to vote for at application startup
    - XXX
fschain_autodeploy: true

network_settings: # NeoFS network settings managed in the Netmap contract
  epoch_duration: 240 # Time interval (approximate) between two adjacent NeoFS epochs measured in Sidechain blocks.
  # Must be an integer in range [1, 18446744073709551615]
  max_object_size: 67108864 # [bytes] Maximum size of physically stored NeoFS objects. Note that this applies
  # only to objects located on storage nodes: user objects have no restrictions and, if necessary, are sliced.
  # Must be an integer in range [1, 18446744073709551615]
  require_homomorphic_hashing: true # Toggles the requirement for homomorphic hashing of object payloads.
  # Must be 'true' or 'false'
  allow_maintenance_mode: true # Toggles permission to transition storage nodes to maintenance state.
  # Must be 'true' or 'false'
  eigen_trust:
    alpha: 0.1 # Alpha parameter of EigenTrust algorithm used in the Reputation system.
    # Must be a floating point number in range [0, 1].
    iterations_number: 4 # Number of EigenTrust algorithm iterations to pass in the Reputation system.
    # Must be an integer in range [1, 18446744073709551615]
  price: # Price settings. NEOFS means NeoFS Balance contract tokens (usually GASe-12).
    storage: 100000000 # [NEOFS] Price for 1GB of data paid every epoch by data owner to storage nodes.
    # Must be an integer in range [0, 18446744073709551615]
    fee:
      ir_candidate: 100 # [GASe-8] Contribution from the new candidate to the Inner Ring. Must be non-negative integer
      # Must be an integer in range [0, 18446744073709551615]
      withdraw: 100000000 # [GASe-8] Fee paid by the user account to;
        # - NeoFS Processing contract (if Notary service is enabled in the NeoFS Mainchain)
      # - each Alphabet member (otherwise)
      # Must be an integer in range [0, 18446744073709551615]
      audit: 10000 # [NEOFS] Fee for data audit paid by storage group owner to the auditor (Inner Ring member).
      # Must be an integer in range [0, 18446744073709551615]
      new_container: 1000 # [NEOFS] Fee for new container paid by creator to each Alphabet member.
      # Must be an integer in range [0, 18446744073709551615]
      container_domain: 500 # [NEOFS] Fee for container's NNS domain paid by container creator to each Alphabet member.
      # Must be a non-negative integer
  custom: # Optional list of custom key-value pairs to be set in the network configuration. Forbidden keys:
    # [AuditFee, BasicIncomeRate, ContainerAliasFee, ContainerFee, EigenTrustAlpha, EigenTrustIterations, EpochDuration,
    # HomomorphicHashingDisabled, InnerRingCandidateFee, MaintenanceModeAllowed, MaxObjectSize, WithdrawFee]
    # Note that this list can be extended in the future, so, to avoid potential collision, it is recommended
    # to use the most specific keys.
    - my_custom_key1=val1
    - my_custom_key2=val2

nns:
  system_email: [email protected]

# Network time settings
timers:
  emit: 50  # Number of sidechain blocks between GAS emission cycles; disabled by default
  stop_estimation:
    mul: 1  # Multiplier in x/y relation of when to stop basic income estimation within the epoch
    div: 4  # Divider in x/y relation of when to stop basic income estimation within the epoch
  collect_basic_income:
    mul: 1  # Multiplier in x/y relation of when to start basic income asset collection within the epoch
    div: 2  # Divider in x/y relation of when to start basic income asset collecting within the epoch
  distribute_basic_income:
    mul: 3  # Multiplier in x/y relation of when to start basic income asset distribution within the epoch
    div: 4  # Divider in x/y relation of when to start basic income asset distribution within the epoch

# Storage node GAS emission settings
emit:
  storage:
    amount: 1000000000  # Fixed8 value of sidechain GAS emitted to all storage nodes once per GAS emission cycle; disabled by default

# Storage node removal settings
netmap_cleaner:
  enabled: true  # Enable voting for removing stale storage nodes from network map
  threshold: 3  # Number of NeoFS epoch without bootstrap request from storage node before it considered stale

# Audit settings
audit:
  pdp:
    max_sleep_interval: 100ms  # Maximum timeout between object.RangeHash requests to the storage node

# Settlement settings
settlement:
  basic_income_rate: 100000000  # Optional: override basic income rate value from network config; applied only in debug mode
  audit_fee: 100000  # Optional: override audit fee value from network config; applied only in debug mode

# LOCODE database
locode:
  db:
    path: /locode/db  # Path to UN/LOCODE database file

After dump-hashes I see new contracts versions (v0.19.1), however hashes are the same as before. Is it a bug?

dump-hashes before update:

(venv.local-pytest) ezayats@testsrv1:~/neofs-dev-env$ ./vendor/neofs-adm morph dump-hashes --config neofs-adm.yml
nns         (v0.17.0):  79d697065a80cb332488c9db1d9b5ba38d9ba0bc  unknown
alphabet 0  (v0.17.0):  34269ea503e495752d62915ca0f154ba7aef92a7  2033-12-05 22:18:26.756 +0000 UTC
audit       (v0.17.0):  3dd128f6ffe307c9fcd2df21f670d39025171234  2033-12-05 22:18:26.756 +0000 UTC
balance     (v0.17.0):  95b7f52b0adde83a0955505d286421a77a168414  2033-12-05 22:18:26.756 +0000 UTC
container   (v0.17.0):  72576c968205e75eabc87708394839f1cf09cd57  2033-12-05 22:18:26.756 +0000 UTC
neofsid     (v0.17.0):  b0aaaac052cc5860b1b79815daae87b3d8d4b849  2033-12-05 22:18:26.756 +0000 UTC
netmap      (v0.17.0):  707516630852f4179af43366917a36b9a78b93a5  2033-12-05 22:18:26.756 +0000 UTC
proxy       (v0.17.0):  f59eb74d61b1df8e1065fe011758a8c55414386d  2033-12-05 22:18:26.756 +0000 UTC
reputation  (v0.17.0):  4b3a8e1e8044affe893855c127fec3880cd5ce76  2033-12-05 22:18:26.756 +0000 UTC

dump-hashes after update:

(venv.local-pytest) ezayats@testsrv1:~/neofs-dev-env$ ./vendor/neofs-adm morph dump-hashes --config neofs-adm.yml
nns         (v0.19.1):  79d697065a80cb332488c9db1d9b5ba38d9ba0bc  unknown
alphabet 0  (v0.19.1):  34269ea503e495752d62915ca0f154ba7aef92a7  2033-12-05 22:18:26.756 +0000 UTC
audit       (v0.19.1):  3dd128f6ffe307c9fcd2df21f670d39025171234  2033-12-05 22:18:26.756 +0000 UTC
balance     (v0.19.1):  95b7f52b0adde83a0955505d286421a77a168414  2033-12-05 22:18:26.756 +0000 UTC
container   (v0.19.1):  72576c968205e75eabc87708394839f1cf09cd57  2033-12-05 22:18:26.756 +0000 UTC
neofsid     (v0.19.1):  b0aaaac052cc5860b1b79815daae87b3d8d4b849  2033-12-05 22:18:26.756 +0000 UTC
netmap      (v0.19.1):  707516630852f4179af43366917a36b9a78b93a5  2033-12-05 22:18:26.756 +0000 UTC
proxy       (v0.19.1):  f59eb74d61b1df8e1065fe011758a8c55414386d  2033-12-05 22:18:26.756 +0000 UTC
reputation  (v0.19.1):  4b3a8e1e8044affe893855c127fec3880cd5ce76  2033-12-05 22:18:26.756 +0000 UTC

After update also managed to get written objects

Logs from updated IR:
updated_ir01.logs.txt

[roman-khimov]

Hashes won't change, that's expected and that's perfectly fine.

[evgeniiz321]

Ok, we can close issue then. Autoupdating was successful.