Improve deployment related code and scripts #896

platfowner · 2022-01-09T08:55:49Z

Change summary:

Finalize state version and write snapshot whenever a block is finalized
Do not write to file for loaded blocks
Extract deleteSnapshot() from writeSnapshot()
In deploy_blockchain_genesis_gcp.sh, kill jobs after file copy and ubuntu package install
In start_node_incremental_gcp.sh, remove old data after killing jobs ([BUG] Failed to load block 0 with --no-keep-data #894)
Replace --canary with BEGIN_PARENT_NODE_INDEX and END_PARENT_NODE_INDEX in incremental deploy
Use ~/ain-blockchain instead of home directory itself for deploy
Add --no-keep-code and --no-keep-data options and set default values of deploy options
Require stronger confirmation for mainnet deploy

Related issues:

liayoo · 2022-01-09T13:02:05Z

deploy_blockchain_genesis_gcp.sh

+    printf "Do you want to proceed for $SEASON? Enter [mainnet]: "
+    read CONFIRM
+    printf "\n\n"
+    if [[ ! $CONFIRM = "mainnet" ]]


liayoo · 2022-01-09T13:08:20Z

deploy_blockchain_incremental_gcp.sh

-else
-    deploy_tracker "$NUM_PARENT_NODES"
-    for j in `seq 0 $(( ${NUM_PARENT_NODES} - 1 ))`
+# Tracker server is deployed with BEGIN_PARENT_NODE_INDEX = -1


liayoo · 2022-01-09T13:14:43Z

start_node_genesis_gcp.sh

-KEEP_DATA_OPTION=""
-FULL_SYNC_OPTION=""
-ACCOUNT_INJECTION_OPTION=""
+ACCOUNT_INJECTION_OPTION="--private-key"


Could you change this to just 'private-key'? It doensn't have the '--' prefix since the blockchain node param doesn't have it. I should've added some notes on the conversion..

Thanks for pointing this out. Just tried to export values properly keeping its values in the script like other variables. PTAL

liayoo · 2022-01-09T13:14:56Z

start_node_genesis_gcp.sh

 printf "JSON_RPC_OPTION=$JSON_RPC_OPTION\n"
 printf "REST_FUNC_OPTION=$REST_FUNC_OPTION\n"

 # NOTE(liayoo): Currently this script supports [--keystore|--mnemonic] option only for the parent chain.
-if [[ $ACCOUNT_INJECTION_OPTION != "private_key" ]] && [[ "$SHARD_INDEX" -gt 0 ]]; then
+if [[ $ACCOUNT_INJECTION_OPTION != "--private_key" ]] && [[ "$SHARD_INDEX" -gt 0 ]]; then


liayoo · 2022-01-09T13:15:41Z

start_node_incremental_gcp.sh

-KEEP_DATA_OPTION=""
-FULL_SYNC_OPTION=""
-ACCOUNT_INJECTION_OPTION=""
+ACCOUNT_INJECTION_OPTION="--private-key"


liayoo · 2022-01-09T13:16:10Z

start_node_incremental_gcp.sh

-  # for performance test pipeline
-  export ENABLE_EXPRESS_RATE_LIMIT=false
+# NOTE(liayoo): Currently this script supports [--keystore|--mnemonic] option only for the parent chain.
+if [[ $ACCOUNT_INJECTION_OPTION != "--private_key" ]] && [[ "$SHARD_INDEX" -gt 0 ]]; then


platfowner · 2022-01-09T14:30:03Z

Addressed the comments so far, PTAL~ @liayoo

liayoo · 2022-01-09T15:16:24Z

start_node_genesis_gcp.sh

@@ -100,18 +95,24 @@ if [[ "$ACCOUNT_INJECTION_OPTION" = "" ]]; then
    return 1
 fi

+if [[ $ACCOUNT_INJECTION_OPTION = "--keystore" ]]; then


checked! 👍

liayoo · 2022-01-09T15:21:48Z

Should we test these scripts with different combinations of options? :)

platfowner · 2022-01-09T16:39:56Z

Should we test these scripts with different combinations of options? :)

Thanks for asking. I tried to test it for as many combinations as I can. Basically,
(setup or not) and
(private-key or keystore) * (keep-code or not) * (keep-data or not) * (fast sync or not)
for genesis and incremental.

I think the most important combinations are
(setup or not) * (keystore) * (keep-code or not) * (keep-data) * (fast sync)

…NODE_INDEX inclusive

liayoo

LGTM, thanks for the nice refactoring!!

liayoo · 2022-01-10T01:22:09Z

Should we test these scripts with different combinations of options? :)

Thanks for asking. I tried to test it for as many combinations as I can. Basically, (setup or not) and (private-key or keystore) * (keep-code or not) * (keep-data or not) * (fast sync or not) for genesis and incremental.

I think the most important combinations are (setup or not) * (keystore) * (keep-code or not) * (keep-data) * (fast sync)

Wow, is that like 64 combinations?! That's really impressive.. Thanks for testing so extensively!

Ah, not actually all of them :) individually and most correlated cases (~ 10 cases).

platfowner · 2022-01-10T02:57:09Z

Thanks for the review! Now merging..

platfowner added bug Something isn't working enhancement New feature or request labels Jan 9, 2022

platfowner requested review from liayoo, minsulee2, ehgmsdk20 and cshcomcom January 9, 2022 08:55

platfowner force-pushed the feature/platfowner/feature branch 2 times, most recently from becedca to c833f94 Compare January 9, 2022 10:14

liayoo reviewed Jan 9, 2022

View reviewed changes

platfowner force-pushed the feature/platfowner/feature branch from 3686889 to 56d60d9 Compare January 9, 2022 16:31

platfowner added 16 commits January 10, 2022 02:33

Extract deleteSnapshot() from writeSnapshot()

614370d

Fix undefined variable

6ab0287

Finalize state version and write snapshot whenever block is finalized

a204560

Log messages for block or snapshot overwriting

b540bf5

Fix a typo

33ce0fa

Do not write to file for loaded blocks

9653dd5

Fix undefined variable error

c6dde8d

Kill jobs after file copy and ubuntu package install

db0d1b0

Set default values of deploy options

929bc05

Add BEGIN_PARENT_NODE_INDEX option to incremeental deploy script

c9f287d

Replace --canary with BEGIN_PARENT_NODE_INDEX and END_PARENT_NODE_INDEX

ee587ce

Use ~/ain-blockchain instead of ~ directory for deployment

3a202fc

Require stronger confirmation for mainnet deployment

a98a0ac

Deploy tracker with BEGIN_PARENT_NODE_INDEX = -1 and make END_PARENT_…

8158675

…NODE_INDEX inclusive

Fix an assignment bug and cp instead of mv for keystore files

818b092

Update sandbox script for directory change

3607fe6

platfowner added 5 commits January 10, 2022 02:35

Update tests and fix test config

0a1917d

Export ACCOUNT_INJECTION_OPTION env variable properly

480ec19

Use loop for iterating nodes in genesis deploy script

fd6890f

Use ~/ain-blockchain directory for monitoring deploy

c75c56a

Fix messaging

f374ecc

platfowner force-pushed the feature/platfowner/feature branch from e6e3963 to f374ecc Compare January 9, 2022 17:35

liayoo approved these changes Jan 10, 2022

View reviewed changes

platfowner merged commit 9c208a4 into develop Jan 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve deployment related code and scripts #896

Improve deployment related code and scripts #896

platfowner commented Jan 9, 2022 •

edited

Loading

liayoo Jan 9, 2022

liayoo Jan 9, 2022

liayoo Jan 9, 2022

platfowner Jan 9, 2022

liayoo Jan 9, 2022

platfowner Jan 9, 2022

liayoo Jan 9, 2022

platfowner Jan 9, 2022

liayoo Jan 9, 2022

platfowner Jan 9, 2022

platfowner commented Jan 9, 2022

liayoo Jan 9, 2022

liayoo commented Jan 9, 2022

platfowner commented Jan 9, 2022

liayoo left a comment

liayoo commented Jan 10, 2022 •

edited by platfowner

Loading

platfowner commented Jan 10, 2022

Improve deployment related code and scripts #896

Improve deployment related code and scripts #896

Conversation

platfowner commented Jan 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

platfowner commented Jan 9, 2022

Choose a reason for hiding this comment

liayoo commented Jan 9, 2022

platfowner commented Jan 9, 2022

liayoo left a comment

Choose a reason for hiding this comment

liayoo commented Jan 10, 2022 • edited by platfowner Loading

platfowner commented Jan 10, 2022

platfowner commented Jan 9, 2022 •

edited

Loading

liayoo commented Jan 10, 2022 •

edited by platfowner

Loading