From 7781a17abf4659a0cf9795f7018c0955e64ab65c Mon Sep 17 00:00:00 2001 From: leovct Date: Mon, 6 Nov 2023 15:50:35 +0100 Subject: [PATCH 1/7] chore: `matic-cli` commands and config update --- README.md | 76 ++++++++++++------------- configs/README.md | 2 +- configs/devnet/docker-setup-config.yaml | 36 ++++++++---- 3 files changed, 64 insertions(+), 50 deletions(-) diff --git a/README.md b/README.md index e17f32fb..6a8f08bf 100644 --- a/README.md +++ b/README.md @@ -7,14 +7,19 @@ test and monitor any devnet on AWS stacks from any local system. It currently supports **only** devnets running `v0.3.x` stacks. -The `express-cli` interacts with `terraform` to create a fully working setup on AWS. +The `express-cli` interacts with `terraform` to create a fully working setup on AWS. This setup is composed by a set of `EC2 VM` instances running a specific `ubuntu 22.04 ami`, mounted with `gp3 disks` , -and a `public-subnet` with its `VPC`. +and a `public-subnet` with its `VPC`. In case the infrastructure already exists, `matic-cli` can be used as a standalone tool to deploy Polygon stacks on pre-configured VMs. Please, refer to the section of this file you are more interested in (`express-cli` or `matic-cli`) +## Table of contents + +- [`express-cli`](#express-cli) +- [`matic-cli`](#matic-cli) + ## `express-cli` ### Requirements @@ -45,7 +50,7 @@ In case you plan to utilize express-cli for Google Cloud, you will need to make ### Auth Configuration -As a prerequisite, you need to configure authentication on `aws` +As a prerequisite, you need to configure authentication on `aws` This will create the folder `~/.aws` in your system To do so, please run @@ -144,8 +149,8 @@ First off, you need to `--init` terraform on your local machine, by executing th - Creates the desired remote setup, based on the preferences defined in the `.env.devnet` file - `--start` command can be used also to target an existing AWS setup. If changes to `.env.devnet` file are detected, the - previous devnet will be destroyed and a new one created, reusing the same AWS VMs - To destroy the remote devnet, you can execute the `--destroy` command. + previous devnet will be destroyed and a new one created, reusing the same AWS VMs + To destroy the remote devnet, you can execute the `--destroy` command. - `../../bin/express-cli --destroy` @@ -225,7 +230,7 @@ The `express-cli` also comes with additional utility commands, listed below. Som - ` ../../bin/express-cli --monitor [exit]` - Monitors the reception of state-syncs and checkpoints to make sure the whole network is in a healthy state. - If `--send-state-sync` hasn't been used before, only checkpoints will be detected. Monitor the setup. + If `--send-state-sync` hasn't been used before, only checkpoints will be detected. Monitor the setup. If `exit` string is passed the process terminates when at least one `stateSync` and one `checkpoint` are detected. - ` ../../bin/express-cli --instances-stop` @@ -316,22 +321,22 @@ Please, make sure to install the following software/packages on the VMs. - Build Essentials (_host_ and _remotes_) ```bash - sudo apt update - sudo apt install build-essential + sudo apt update --yes && sudo apt install --yes build-essential ``` - Go 1.18+ (_host_ and _remotes_) ```bash - wget https://raw.githubusercontent.com/maticnetwork/node-ansible/master/go-install.sh - bash go-install.sh --remove - bash go-install.sh + wget https://raw.githubusercontent.com/maticnetwork/node-ansible/master/go-install.sh \ + && chmod +x go-install.sh \ + && bash go-install.sh --remove \ + && bash go-install.sh ``` - Rabbitmq (_host_ and _remotes_) ```bash - sudo apt install rabbitmq-server + sudo apt install --yes rabbitmq-server ``` - Docker (_host_ and _remotes_, only needed in case of a docker setup) @@ -342,32 +347,25 @@ Please, make sure to install the following software/packages on the VMs. - Node v16.17.1 (only _host_) ```bash - curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash - nvm install 16.17.1 + curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash && nvm install 16.17.1 ``` - Npm (only _host_) ```bash - sudo apt update - sudo apt install nodejs npm + sudo apt update --yes && sudo apt install --yes npm ``` - Python 2 (only _host_) ```bash - sudo apt install python2 && alias python="/usr/bin/python2 + sudo apt install python2 --yes && alias python="/usr/bin/python2" ``` -- Solc v0.5.16 (only _host_) - - ```bash - sudo snap install solc - ``` +- Solc (solidity compiler) and Ganache CLI (only _host_) -- Ganache CLI (only _host_) ```bash - sudo npm install -g ganache + npm install --global solc ganache ``` ### Usage @@ -375,28 +373,28 @@ Please, make sure to install the following software/packages on the VMs. On the _host_ machine, please run ```bash -cd ~ -git clone https://github.com/maticnetwork/matic-cli.git -cd matic-cli -npm i -mkdir devnet -cd devnet +cd \ + && git clone https://github.com/maticnetwork/matic-cli.git \ + && cd matic-cli \ + && npm install ``` +Set up this handy alias: `alias matic-cli="$(pwd)/bin/matic-cli"`. + #### Local dockerized network Adjust the [docker configs](configs/devnet/docker-setup-config.yaml) and run ```bash -../bin/matic-cli setup devnet -c ../configs/devnet/docker-setup-config.yaml +mkdir devnet \ + && cd devnet \ + && matic-cli setup devnet --config ../configs/devnet/docker-setup-config.yaml | tee setup.log +... +DONE Devnet is ready ``` Once the setup is done, follow these steps for local docker deployment -- Move to devnet folder - ```bash - cd matic-cli/devnet - ``` - Start ganache ```bash @@ -441,16 +439,16 @@ Note: in case of docker setup, we have provided [some additional scripts](src/se Adjust the [remote configs](configs/devnet/remote-setup-config.yaml) and run ```bash -../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml +matic-cli setup devnet --config ../configs/devnet/remote-setup-config.yaml ``` Alternatively, this step can be executed interactively with ```bash -../bin/matic-cli setup devnet -i +matic-cli setup devnet --interactive ``` -Once the setup is done, follow these steps for remote deployment +Once the setup is done, follow these steps for remote deployment In this case, the stack is already running, you would just need to deploy/sync some contracts, as follows: - Move to devnet folder @@ -481,7 +479,7 @@ Stop al services, remove the `matic-cli/devnet` folder, and you can start the pr eval "$(ssh-agent -s)" ssh-add `<.pem file>` ``` -3. We have provided the default config values [here](configs/devnet) to ensure smooth functioning of the process +3. We have provided the default config values [here](configs/devnet) to ensure smooth functioning of the process Please check the relative [README](configs/README.md) for more accurate description of such configs These files are used as templates and dynamically modified by `express-cli`, hence they should not be deleted nor any modification remotely pushed Therefore, they are under `.gitignore`, and in case you do not want those changes to be reflected in your local `git`, diff --git a/configs/README.md b/configs/README.md index 9db49180..4a36ec41 100644 --- a/configs/README.md +++ b/configs/README.md @@ -7,7 +7,7 @@ For `mati-cli`, the configs are directly used to spin the network up. ## Usage ```bash -matic-cli setup devnet -c path/to/config.yaml +matic-cli setup devnet --config path/to/config.yaml ``` ## Option details with examples diff --git a/configs/devnet/docker-setup-config.yaml b/configs/devnet/docker-setup-config.yaml index 0282df57..cce99eb3 100755 --- a/configs/devnet/docker-setup-config.yaml +++ b/configs/devnet/docker-setup-config.yaml @@ -1,15 +1,31 @@ -defaultStake: 10000 -defaultFee: 2000 -borChainId: 15005 -heimdallChainId: heimdall-4052 +# Repositories borRepo: https://github.com/maticnetwork/bor.git borBranch: develop + heimdallRepo: https://github.com/maticnetwork/heimdall.git heimdallBranch: develop + contractsRepo: https://github.com/maticnetwork/contracts.git contractsBranch: mardizzone/node-16 + genesisContractsRepo: https://github.com/maticnetwork/genesis-contracts.git genesisContractsBranch: master + +# Defaults +devnetType: docker +defaultStake: 10000 +defaultFee: 2000 + +# L1 +ethURL: http://ganache:9545 +ethHostUser: ubuntu + +# Heimdall +heimdallChainId: heimdall-4052 +heimdallDockerBuildContext: https://github.com/maticnetwork/heimdall.git#develop + +# Bor +borChainId: 15005 sprintSize: 64 sprintSizeBlockNumber: 0 blockNumber: '0' @@ -17,11 +33,11 @@ blockTime: '2' numOfBorValidators: 1 numOfBorSentries: 1 numOfBorArchiveNodes: 0 -ethURL: http://ganache:9545 -ethHostUser: ubuntu -devnetType: docker borDockerBuildContext: https://github.com/maticnetwork/bor.git#develop -heimdallDockerBuildContext: https://github.com/maticnetwork/heimdall.git#develop devnetBorUsers: ubuntu,ubuntu -devnetBorHosts: - - ec2-xx-xxx-xx-xxx.eu-west-1.compute.amazonaws.com # use localhost for local deployments +devnetBorHosts: ["ec2-xx-xxx-xx-xxx.eu-west-1.compute.amazonaws.com"] # use localhost for local deployments + +# Erigon +numOfErigonValidators: 0 +numOfErigonSentries: 0 +numOfErigonArchiveNodes: 0 From 537558e11e73f948049dbb286f0c4698451de6b9 Mon Sep 17 00:00:00 2001 From: leovct Date: Wed, 8 Nov 2023 13:05:05 +0100 Subject: [PATCH 2/7] chore: nit --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 6a8f08bf..882babab 100644 --- a/README.md +++ b/README.md @@ -328,7 +328,6 @@ Please, make sure to install the following software/packages on the VMs. ```bash wget https://raw.githubusercontent.com/maticnetwork/node-ansible/master/go-install.sh \ - && chmod +x go-install.sh \ && bash go-install.sh --remove \ && bash go-install.sh ``` From 606e588c29c5cd05e4c5b7de03add7473d8fd02a Mon Sep 17 00:00:00 2001 From: leovct Date: Wed, 8 Nov 2023 13:12:28 +0100 Subject: [PATCH 3/7] ci: few nits --- .github/workflows/ci.yml | 44 +++++++++++++++++----------------------- 1 file changed, 19 insertions(+), 25 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bf9b26b1..ceb0b25e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -28,19 +28,16 @@ jobs: node-version: '16.17.1' - name: Install npm dependencies - run: | - cd matic-cli - npm install --prefer-offline --no-audit --progress=false + working-directory: matic-cli + run: npm install --prefer-offline --no-audit --progress=false - name: Run prettier - run: | - cd matic-cli - npm run prettier:check + working-directory: matic-cli + run: npm run prettier:check - name: Run lint - run: | - cd matic-cli - npm run lint:check + working-directory: matic-cli + run: npm run lint:check integration-tests: permissions: @@ -57,8 +54,8 @@ jobs: path: matic-cli - name: Create .env file + working-directory: matic-cli run: | - cd matic-cli cp .env.example .env sed -i 's,YOUR_IDENTIFIER,matic-cli-ci,' .env sed -i 's,aws-key,matic-cli-ci-key,' .env @@ -70,8 +67,8 @@ jobs: uses: haythem/public-ip@v1.3 - name: Create secret.tfvars + working-directory: matic-cli run: | - cd matic-cli touch secret.tfvars echo "SG_CIDR_BLOCKS=[\"${{ steps.ip.outputs.ipv4 }}/32\"]" >> secret.tfvars @@ -83,8 +80,8 @@ jobs: role-session-name: GithubActionsSession - name: Create aws key pair + working-directory: matic-cli run: | - cd matic-cli mkdir aws cd aws aws ec2 create-key-pair --key-name matic-cli-ci-key --key-type rsa --key-format pem --query "KeyMaterial" --output text > matic-cli-ci-key.pem @@ -96,30 +93,28 @@ jobs: node-version: '16.17.1' - name: Install npm dependencies - run: | - cd matic-cli - npm install --prefer-offline --no-audit --progress=false + working-directory: matic-cli + run: npm install --prefer-offline --no-audit --progress=false - name: Init devnet - run: | - cd matic-cli - ./bin/express-cli --init aws + working-directory: matic-cli + run: ./bin/express-cli --init aws - name: Start devnet + working-directory: matic-cli/deployments/devnet-1 run: | - cd matic-cli/deployments/devnet-1 ls -la ../../bin/express-cli --start - name: Run stateSynced and checkpoint tests + working-directory: matic-cli/deployments/devnet-1 run: | - cd matic-cli/deployments/devnet-1 ../../bin/express-cli --send-state-sync timeout 20m ../../bin/express-cli --monitor exit - name: Run smart contracts events tests + working-directory: matic-cli/deployments/devnet-1 run: | - cd matic-cli/deployments/devnet-1 timeout 5m ../../bin/express-cli --send-staked-event 1 timeout 5m ../../bin/express-cli --send-stakeupdate-event 1 timeout 5m ../../bin/express-cli --send-topupfee-event 1 @@ -127,13 +122,12 @@ jobs: - name: Destroy devnet if: always() + working-directory: matic-cli/deployments/devnet-1 run: | - cd matic-cli/deployments/devnet-1 echo "Running --destroy" ../../bin/express-cli --destroy - name: Delete aws key pair if: always() - run: | - cd matic-cli/aws - aws ec2 delete-key-pair --key-name matic-cli-ci-key + working-directory: matic-cli/aws + run: aws ec2 delete-key-pair --key-name matic-cli-ci-key From 0d10029e4042d41c93cbb08184d8edd9c3e350a1 Mon Sep 17 00:00:00 2001 From: leovct Date: Mon, 13 Nov 2023 17:34:47 +0100 Subject: [PATCH 4/7] doc: add handy cmd to retrieve addresses --- src/setup/devnet/templates/docker/README.md | 58 +++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/src/setup/devnet/templates/docker/README.md b/src/setup/devnet/templates/docker/README.md index 67a63b2c..ab7deca7 100644 --- a/src/setup/devnet/templates/docker/README.md +++ b/src/setup/devnet/templates/docker/README.md @@ -25,6 +25,64 @@ On docker container's shell: $ tail -f /root/heimdall/logs/heimdalld.log ``` +**To retrieve nodes and contracts addresses** + +After starting bor nodes, you can retrieve the public and private keys of the node. + +```bash +# example with the first node (node0). +$ cat devnet/devnet/node0/bor/address.txt +0x2Bd3C50030325a0587f6c9AE3d3Aa5C2b14b72E6 + +$ cat devnet/devnet/node0/bor/privatekey.txt +0x2bf046c818b06eca93f25dcb08ff89658bd1acd54c43d6e1fffe78c9332fd50a +``` + +After starting the chains and deploying contracts on the child chain, you can retrieve the contract addresses. + +```bash +$ cat devnet/code/contracts/contractAddresses.json +{ + "root": { + "Registry": "0xB39aA4E9646Fd9C50ee8C871Fe345569C4D5D04A", + "RootChain": "0x67D07acbA0389ECF514eF8B43dFE4D8f6dDBB5d2", + "GovernanceProxy": "0x0e63099e1420C7a3BF8C226730eEDc6C2d1A1704", + "RootChainProxy": "0xBa1F4D93D62c24dAE67c33F9Cd1bECaa2d46dD5B", + "DepositManager": "0x5Bc2B859D594e136A0552e7BDF2dF3867424Cce7", + "DepositManagerProxy": "0x4A356C9CAeDfefE43c144419487e06c04296bf69", + "WithdrawManager": "0x765fd35DBbDff1dd36e422C76Fa37a6F55A4687d", + "WithdrawManagerProxy": "0x50415F3215599Bd6B92bA68cea61b46A15022F59", + "StakeManager": "0x09F8d80E3d162CD8CBbD494633b9D3eD75bDF36e", + "StakeManagerProxy": "0x9Db18c0C0384c2986F70e8B109a564129599Ce23", + "SlashingManager": "0xF67b5A1A85b4e50F594d450dB2a2f2FCfD24DEF0", + "StakingInfo": "0xcC589091De6aE35b53dF660A347c3887E6C94461", + "ExitNFT": "0x29A4a587c0642B1234b2918Fb37Ca06270D1a1CF", + "StateSender": "0xF88d3899fD32f0531E100d258160738145379E14", + "predicates": { + "ERC20Predicate": "0x25405947a3bD2BD7767C2Cf8124F315883fD7111", + "ERC721Predicate": "0xBe710e1C3F7F4c7E295A4686d1d6478512238EA4", + "MarketplacePredicate": "0xe3c38Ab717c4e489C602196Cc2902F7AfeD3765D", + "TransferWithSigPredicate": "0x497506B0Be50dcD034b03D6565fc0A39EC513092" + }, + "tokens": { + "MaticToken": "0x4b7FF52bbe91CF6A743D428D84F38162ea3a1310", + "MaticWeth": "0xc888E865D9067E2c44672Fa50bE2B77a781528EC", + "TestToken": "0x2DDAB92De80d1D9D7e89fA49429F88d47d7222Bc", + "RootERC721": "0xe80F298C455A5aD3EbB8B6ab602A336a43C958A7" + } + }, + "child": { + "ChildChain": "0x680C2C3f42213FA92868b80E0aCb6430850186CE", + "tokens": { + "MaticWeth": "0xd30123F2B51a9C043755cAF07a833124933B1edb", + "MaticToken": "0x0000000000000000000000000000000000001010", + "TestToken": "0xe249c8135a138fF48C8b79Ef7ABe8c0C902cC298", + "RootERC721": "0x43e5D5380B8D5FA8aA1fe67D0C3F33D5B0FcE26b" + } + } +} +``` + ### Run multiple bor nodes ```bash From 5181c12147d442197a34e54a640a11c0d93d405d Mon Sep 17 00:00:00 2001 From: leovct Date: Tue, 14 Nov 2023 09:30:21 +0100 Subject: [PATCH 5/7] chore: pr review --- README.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 882babab..6ff8d4ce 100644 --- a/README.md +++ b/README.md @@ -378,8 +378,6 @@ cd \ && npm install ``` -Set up this handy alias: `alias matic-cli="$(pwd)/bin/matic-cli"`. - #### Local dockerized network Adjust the [docker configs](configs/devnet/docker-setup-config.yaml) and run @@ -387,13 +385,19 @@ Adjust the [docker configs](configs/devnet/docker-setup-config.yaml) and run ```bash mkdir devnet \ && cd devnet \ - && matic-cli setup devnet --config ../configs/devnet/docker-setup-config.yaml | tee setup.log + && ../bin/matic-cli setup devnet --config ../configs/devnet/docker-setup-config.yaml | tee setup.log ... DONE Devnet is ready ``` Once the setup is done, follow these steps for local docker deployment +- Move to devnet folder + + ```bash + cd matic-cli/devnet + ``` + - Start ganache ```bash @@ -438,13 +442,13 @@ Note: in case of docker setup, we have provided [some additional scripts](src/se Adjust the [remote configs](configs/devnet/remote-setup-config.yaml) and run ```bash -matic-cli setup devnet --config ../configs/devnet/remote-setup-config.yaml +../bin/matic-cli setup devnet --config ../configs/devnet/remote-setup-config.yaml ``` Alternatively, this step can be executed interactively with ```bash -matic-cli setup devnet --interactive +../bin/matic-cli setup devnet --interactive ``` Once the setup is done, follow these steps for remote deployment From 26088f38f008895c3c1d5801d6799c6eed13d5ef Mon Sep 17 00:00:00 2001 From: leovct Date: Tue, 14 Nov 2023 09:32:20 +0100 Subject: [PATCH 6/7] chore: nit --- README.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/README.md b/README.md index 6ff8d4ce..af0733de 100644 --- a/README.md +++ b/README.md @@ -392,12 +392,6 @@ DONE Devnet is ready Once the setup is done, follow these steps for local docker deployment -- Move to devnet folder - - ```bash - cd matic-cli/devnet - ``` - - Start ganache ```bash From 618286599bf6536dc208f3ea71f42c62a0738fc4 Mon Sep 17 00:00:00 2001 From: leovct Date: Tue, 14 Nov 2023 10:02:06 +0100 Subject: [PATCH 7/7] chore: pr comments --- README.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index af0733de..e18e332f 100644 --- a/README.md +++ b/README.md @@ -346,7 +346,9 @@ Please, make sure to install the following software/packages on the VMs. - Node v16.17.1 (only _host_) ```bash - curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash && nvm install 16.17.1 + curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash \ + && source /home/ubuntu/.bashrc \ + && nvm install 16.17.1 ``` - Npm (only _host_) @@ -361,10 +363,16 @@ Please, make sure to install the following software/packages on the VMs. sudo apt install python2 --yes && alias python="/usr/bin/python2" ``` -- Solc (solidity compiler) and Ganache CLI (only _host_) +- Solc v0.5.16 (only _host_) ```bash - npm install --global solc ganache + sudo snap install solc + ``` + +- Ganache CLI (only _host_) + + ```bash + npm install --global ganache ``` ### Usage