From d0c2a4859f66a8a50eff2f64915d5c53c24910f3 Mon Sep 17 00:00:00 2001 From: syntrust Date: Mon, 4 Mar 2024 18:31:01 -0800 Subject: [PATCH 1/5] update dashboard link --- GUIDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/GUIDE.md b/GUIDE.md index acfee507..cbe48bfc 100644 --- a/GUIDE.md +++ b/GUIDE.md @@ -242,7 +242,7 @@ INFO [01-19|14:41:49.018] Mining transaction accounting (in ether) reward=0.0101 ``` -You can also visit [the EthStorage dashboard](http://grafana.ethstorage.io/d/es-node-mining-state-dev/ethstorage-monitoring-dev?orgId=2&refresh=5m) for real-time mining statistics. +You can also visit [the EthStorage dashboard](https://grafana.ethstorage.io/d/es-node-mining-state-sepolia/ethstorage-monitoring-sepolia?orgId=2&refresh=5m&var-Miner=All) for real-time mining statistics. Finally, pay attention to the balance change of your miner's address which reflects the mining income. From d67f9ff66f013215de3afd0bd8d6fdddf1be6b73 Mon Sep 17 00:00:00 2001 From: syntrust Date: Mon, 4 Mar 2024 18:35:31 -0800 Subject: [PATCH 2/5] update dashboard link --- GUIDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/GUIDE.md b/GUIDE.md index cbe48bfc..87875d34 100644 --- a/GUIDE.md +++ b/GUIDE.md @@ -242,7 +242,7 @@ INFO [01-19|14:41:49.018] Mining transaction accounting (in ether) reward=0.0101 ``` -You can also visit [the EthStorage dashboard](https://grafana.ethstorage.io/d/es-node-mining-state-sepolia/ethstorage-monitoring-sepolia?orgId=2&refresh=5m&var-Miner=All) for real-time mining statistics. +You can also visit [the EthStorage dashboard](https://grafana.ethstorage.io/) for real-time mining statistics. Finally, pay attention to the balance change of your miner's address which reflects the mining income. From 3aaff6a67c0d1b0e1b26582551db13aa30e66e4e Mon Sep 17 00:00:00 2001 From: syntrust Date: Tue, 5 Mar 2024 10:49:30 -0800 Subject: [PATCH 3/5] update readme to be aligned with docs --- GUIDE.md | 267 ------------------------------------------------------ README.md | 128 ++------------------------ SETUP.md | 229 ---------------------------------------------- 3 files changed, 8 insertions(+), 616 deletions(-) delete mode 100644 GUIDE.md delete mode 100644 SETUP.md diff --git a/GUIDE.md b/GUIDE.md deleted file mode 100644 index 87875d34..00000000 --- a/GUIDE.md +++ /dev/null @@ -1,267 +0,0 @@ -# Es-node Quick Start - -This tutorial provides practical steps to start an es-node instance for connecting to the existing EthStorage devnet. - -For setting up a new private EthStorage testnet, please refer to this [guide](/SETUP.md). - -For a detailed explanation for es-node please consult the [README](/README.md). - -## Testnet spec - -- Layer 1: Sepolia testnet -- storage-contracts-v1: v0.1.0 -- es-node: v0.1.6 - -## Minimum Hardware Requirements - -Please refer to [this section](/README.md/#minimum-hardware-requirements) for hardware requirements. - -## System Environment - - Ubuntu 20.04+ (tested and verified) - - (Optional) Docker 24.0.5+ (would simplify the process) - - (Optional) Go 1.20+ and Node.js 16+ (can be installed following the [steps](#install-dependencies)) - -You can choose [the method of running es-node](#options-to-run-es-node) based on your current environment. - -_Note: The steps assume the use of the root user for all command line operations. If using a non-root user, you may need to prepend some commands with "sudo"._ - -## Preparing miner and signer account - -It is recommended to prepare two Ethereum accounts specifically for this test. One of these accounts should contain a balance of test ETH to be used as a transaction signer. - -The test ETH can be requested from [https://sepoliafaucet.com/](https://sepoliafaucet.com/). - -Remember to use the signer's private key (with ETH balance) to replace `` in the following steps. And use the other address to replace ``. - -## Options for running es-node - -You can run es-node from a pre-built Docker image, a pre-built executable, or from the source code. - - - If you choose [the pre-built es-node executable](#from-pre-built-executables), you will need to manually install some dependencies such as Node.js and snarkjs. - - - If you have Docker version 24.0.5 or above installed, the quickest way to get started is by [using a pre-built Docker image](#from-a-docker-image). - - - If you prefer to build [from the source code](#from-source-code), you will also need to install Go besides Node.js and snarkjs. - -## From pre-built executables - -Before running es-node from the pre-built executables, ensure that you have installed [Node.js](#install-nodejs), [snarkjs](#install-snarkjs) and [WSL](https://learn.microsoft.com/en-us/windows/wsl/install) if you are on Windows. - -Download the pre-built package suitable for your platform: - -Linux x86-64 or WSL (Windows Subsystem for Linux): -```sh -curl -L https://github.com/ethstorage/es-node/releases/download/v0.1.6/es-node.v0.1.6.linux-amd64.tar.gz | tar -xz -``` -MacOS x86-64: -```sh -curl -L https://github.com/ethstorage/es-node/releases/download/v0.1.6/es-node.v0.1.6.darwin-amd64.tar.gz | tar -xz -``` -MacOS ARM64: -```sh -curl -L https://github.com/ethstorage/es-node/releases/download/v0.1.6/es-node.v0.1.6.darwin-arm64.tar.gz | tar -xz -``` -Run es-node -``` -cd es-node.v0.1.6 -env ES_NODE_STORAGE_MINER= ES_NODE_SIGNER_PRIVATE_KEY= ./run.sh -``` - -## From a Docker image - -Run an es-node container in one step: -```sh -docker run --name es -d \ - -v ./es-data:/es-node/es-data \ - -e ES_NODE_STORAGE_MINER= \ - -e ES_NODE_SIGNER_PRIVATE_KEY= \ - -p 9545:9545 \ - -p 9222:9222 \ - -p 30305:30305/udp \ - --entrypoint /es-node/run.sh \ - ghcr.io/ethstorage/es-node:v0.1.6 -``` - -You can check docker logs using the following command: -```sh -docker logs -f es -``` -## From source code - -You will need to [install Go](#install-go) to build es-node from source code, and install [Node.js](#install-nodejs) and [snarkjs](#install-snarkjs) to run es-node. - -Download source code and switch to the latest release branch: -```sh -git clone https://github.com/ethstorage/es-node.git -cd es-node -git checkout v0.1.6 -``` -Build es-node: -```sh -make -``` - -Start es-node -```sh -chmod +x run.sh && env ES_NODE_STORAGE_MINER= ES_NODE_SIGNER_PRIVATE_KEY= ./run.sh -``` - -With source code, you also have the option to build a Docker image by yourself and run an es-node container: - -```sh -env ES_NODE_STORAGE_MINER= ES_NODE_SIGNER_PRIVATE_KEY= docker-compose up -``` -If you want to run Docker container in the background and keep all the logs: -```sh -env ES_NODE_STORAGE_MINER= ES_NODE_SIGNER_PRIVATE_KEY= ./run-docker.sh -``` - - -## Install dependencies - -_Please note that not all steps in this section are required; they depend on your [choice](#options-for-running-es-node)._ - -### Install Go - -Download a stable Go release, e.g., go1.21.4. -```sh -curl -OL https://golang.org/dl/go1.21.4.linux-amd64.tar.gz -``` -Extract and install -```sh -tar -C /usr/local -xf go1.21.4.linux-amd64.tar.gz -``` -Update `$PATH` -``` -echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.profile -source ~/.profile -``` - -### Install Node.js - -Install Node Version Manager -```sh -curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash -``` -Close and reopen your terminal to start using nvm or run the following to use it now: -```sh -export NVM_DIR="$HOME/.nvm" -[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm -[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion -``` -Choose a Node.js version above 16 (e.g. v20.*) and install -```sh -nvm install 20 -``` -Activate the Node.js version -```sh -nvm use 20 -``` -### Install snarkjs -```sh -npm install -g snarkjs -``` - -## Two phases after es-node launch - -After the launch of ES node, it basically goes through two main stages. - -### Data sync phase - -The es-node will synchronize data from other peers in the network. You can check from the console the number of peers the node is connected to and, more importantly, the estimated syncing time. - -During this phase, the CPUs are expected to be fully occupied for data processing. If not, please refer to [the FAQ](#how-to-tune-the-performance-of-syncing) for performance tuning on this area. - -A typical log entry in this phase appears as follows: - -``` -INFO [01-18|09:13:32.564] Storage sync in progress progress=85.50% peerCount=2 syncTasksRemain=1@0 blobsSynced=1@128.00KiB blobsToSync=0 fillTasksRemain=30 emptyFilled=3,586,176@437.77GiB emptyToFill=608,127 timeUsed=1h48m7.556s eta=18m20.127s - -``` -### Sampling phase - -Once data synchronization is complete, the es-node will enter the sampling phase, also known as mining. - -A typical log entry of sampling during a slot looks like this: - -``` -INFO [01-19|05:02:23.210] Mining info retrieved shard=0 LastMineTime=1,705,634,628 Difficulty=4,718,592,000 proofsSubmitted=6 -INFO [01-19|05:02:23.210] Mining tasks assigned shard=0 threads=64 block=10,397,712 nonces=1,048,576 -INFO [01-19|05:02:26.050] The nonces are exhausted in this slot, waiting for the next block samplingTime=2.8s shard=0 block=10,397,712 - -``` -When you see "The nonces are exhausted in this slot...", it indicates that your node has successfully completed all the sampling tasks within a slot. -The "samplingTime" value informs you of the duration, in seconds, it took to complete the sampling process. - -If the es-node doesn't have enough time to complete sampling within a slot, the log will display "Mining tasks timed out". For further actions, please refer to [the FAQ](#what-can-i-do-about-mining-tasks-timed-out). - -## FAQ - -### What should I do when the log frequently shows "i/o deadline reached" during data syncing? - -The timeout issue could be due to network reasons. It may be because the amount of data requested relative to network performance is too large, so even a slight delay in individual requests will be magnified. - -You can change the size of each request to a smaller value using `--p2p.max.request.size`. -The default value is `4194304`. You can try adjusting it to `1048576`. - -### How to tune the performance of syncing? - -To enhance syncing performance or limit the usage of CPU power, you can adjust the values of the following related flags with higher or lower values: - -- The `--p2p.sync.concurrency` flag determines the number of threads that simultaneously retrieve shard data, with a default setting of 16. -- The `--p2p.fill-empty.concurrency` flag specifies the number of threads used to concurrently fill encoded empty blobs, with a default setting of the CPU number minus 2. - -### Why is the CPU fully utilized when syncing data? Does this imply that running an es-node requires high-end hardware? Additionally, is there a way to configure it to use less CPU power? I'm asking because I need to run other software on the same machine. - -During data synchronization, the CPUs are fully utilized as the es-node is engaged in intensive data processing tasks like encoding and writing. - -However, this high-intensity processing occurs primarily when large volumes of data need to be synchronized, such as during the initial startup of the es-node. Once the synchronization is complete, and the es-node enters the mining phase, the CPU usage decreases significantly. So, generaly speaking, the es-node does not require high-end hardware, particularly for the CPU. - -If there is a need to conserve CPU power anyway, you can tune down the values of syncing performance related flags, namely `--p2p.sync.concurrency` and `--p2p.fill-empty.concurrency`. See [here](#how-to-tune-the-performance-of-syncing) for detailed information. - -### What does it means when the log shows "The nonces are exhausted in this slot" - -When you see "The nonces are exhausted in this slot...", it indicates that your node has successfully completed all the sampling tasks within a slot, and you do not need to do anything about it. - -### What can I do about "Mining tasks timed out"? - -When you see the message "Mining tasks timed out", it indicates that the sampling task couldn't be completed within a slot. - -You can check the IOPS of the disk to determine if the rate of data read has reached the IO capacity. -If not, using the flag `--miner.threads-per-shard` can specify the number of threads to perform sampling for each shard, thereby helping in accomplishing additional sampling. - -### How do I know whether I've got a mining reward? - -You can observe the logs to see if there are such messages indicating a successful storage proof submission: - -``` -INFO [01-19|14:41:48.715] Mining transaction confirmed txHash=62df87..7dbfbc -INFO [01-19|14:41:49.017] "Mining transaction success! √" miner=0x534632D6d7aD1fe5f832951c97FDe73E4eFD9a77 -INFO [01-19|14:41:49.018] Mining transaction details txHash=62df87..7dbfbc gasUsed=419,892 effectiveGasPrice=2,000,000,009 -INFO [01-19|14:41:49.018] Mining transaction accounting (in ether) reward=0.01013071059 cost=0.0008397840038 profit=0.009290926583 - -``` - -You can also visit [the EthStorage dashboard](https://grafana.ethstorage.io/) for real-time mining statistics. - -Finally, pay attention to the balance change of your miner's address which reflects the mining income. - -### Can I change the data file location? - -Yes, you can. - -By default, when executed, the run.sh script will generate a data file named `shard-0.dat` to store shard 0 in the data directory specified by `--datadir`, located in the same folder as the run.sh script. - -If necessary, you can choose an alternative location for data storage by specifying the full path of the file as the value of the `--storage.files` flag in the run.sh script. - -Please refer to [configuration](/README.md#configuration) for more details. - -### What can I do about "The zkey file was not downloaded" error? - -When you see the following message when running **run.sh**. you can manually download the [**blob_poseidon.zkey**](https://drive.google.com/file/d/1ZLfhYeCXMnbk6wUiBADRAn1mZ8MI_zg-/view) / [**blob_poseidon2.zkey**](https://drive.google.com/file/d/1olfJvXPJ25Rbcjj9udFlIVr08cUCgE4l/view) to `./build/bin/snarkjs/` folder and run it again. -``` -zk prover mode is 2 -Start downloading ./build/bin/snarkjs/blob_poseidon2.zkey... -... ... -Error: The zkey file was not downloaded. Please try again. -``` \ No newline at end of file diff --git a/README.md b/README.md index d7caeeb8..204ca6a5 100644 --- a/README.md +++ b/README.md @@ -1,130 +1,18 @@ -# es-node Golang implementation of the EthStorage node. -EthStorage is a storage-specific L2 network that reuses Ethereum security to extend Ethereum storage capabilities via data availability technology. +# What is EthStorage? -Storage providers can join the L2 network permissionlessly by running an es-node. They need to download the replica of the L2 data into their local node, submit [proof of storage](#about-proof-of-storage) to the L1 EthStorage contract, and get the corresponding reward once the proof is accepted by the contract. +EthStorage is a modular and decentralized storage Layer 2 that offers programmable key-value storage powered by DA. It enables long-term DA solutions for Rollups and opens up new possibilities for fully on-chain applications like games, social networks, AI, etc. -## About Proof of Storage +Check [the Documentation](https://docs.ethstorage.io/) if you want to know more about EthStorage. -To check if a replica of data is indeed physically stored, the storage providers need to randomly sample the encoded BLOBs with unique storage provider ID (miner address) and submit the proofs to the L1 storage contract for verification over time. That is how storage providers collect their storage fees. +# Getting Started -The continuous random sampling needs to satisfy a difficulty parameter that will be adjusted on-chain similarly to Ethash difficulty adjustment algorithm. So the Proof of Storage process sometimes is referred to as _`mining`_. +If you are interested in earning rewards by storing data, you may want to consider becoming a storage provider in the EthStorage network, a fully permissionless process as simple as getting the es-node up and running. -To get ready to generate and submit the proof of storage, you need to prepare a miner address to generate unique physical replicas and receive storage fees, as well as a private key to sign the transactions that submit the storage proofs. +For prerequisites and detailed instructions to run es-node please read [the Storage Providers' guide](https://docs.ethstorage.io/storage-provider-guide). -## Minimum Hardware Requirements +# License -The minimum hardware requirements for an es-node are as follows: - - - CPU: A minimum of 4 cores and 8 threads - - 4GB of RAM - - Disk: - - We recommend using an NVMe disk to support the full speed of sampling - - At least 550 GB of available storage space for the runtime and sync of one data shard - - Internet service: At least 8MB/sec download speed - -## Getting started - -To start an es-node, you have the option to run a manually built binary or with Docker. The `run.sh` script is used as an entry point in all the above options. The main function of the script is to initialize the data file, prepare for [Proof of Storage](#about-proof-of-storage), and launch es-node with preset parameters. -[Proof of Storage](#about-proof-of-storage) is enabled by default by the `--miner.enabled` flag in `run.sh`, which means you become a storage provider when you start an es-node with default settings. - -Refer to [here](/GUIDE.md) for a quick start to run an es-node to connect to the current EthStorage devnet. - -Alternatively, refer to [here](/SETUP.md) for a quick start to set up your own EIP-4844 devnet and EthStorage devnet. - -_Note: Some of the flags/parameters used in `run.sh` are supposed to change over time. Refer to [configuration](#configuration) for a full list._ - - -## Configuration - -### Configuration to create data files - -With `es-node init` command, you can init your es-node by creating a data file for each shard. - -You can specify `shard_len` (the number of shards) or `shard_index` (the index of specified shards, and you can specify more than one) to create shards that you would like to mine. If both appear, `shard_index` takes precedence. - -Here are the options that you can use with `init` command: - -|Name|Description|Default|Required| -|---|---|---|---| -|`--datadir`|Data directory for the storage files, databases and keystore||✓| -|`--encoding_type`|Encoding type of the shards. 0: no encoding, 1: keccak256, 2: ethash, 3: blob poseidon. Default: 3||| -|`--shard_len`|Number of shards to mine. Will create one data file per shard.||| -|`--shard_index`|Indexes of shards to mine. Will create one data file per shard.||| -|`--l1.rpc`|Address of L1 User JSON-RPC endpoint to use (eth namespace required)||✓| -|`--storage.l1contract`|Storage contract address on l1||✓| -|`--storage.miner`|Miner's address to encode data and receive mining rewards||✓| - -### Configuration to run es-node - -The full list of options that you can use to configure an es-node are as follows: -|Name|Description|Default|Required| -|---|---|---|---| -|`--datadir`|Data directory for the storage files, databases and keystore||✓| -|`--download.dump`|Where to dump the downloaded blobs||| -|`--download.start`|Block number which the downloader download blobs from|`0`|| -|`--l1.beacon`|Address of L1 beacon chain endpoint to use||✓| -|`--l1.beacon-based-slot`|A slot number in the past time specified by l1.beacon-based-time|`0`|✓| -|`--l1.beacon-based-time`|Timestamp of a slot specified by l1.beacon-based-slot|`0`|✓| -|`--l1.beacon-slot-time`|Slot time of the L1 beacon chain|`12`|| -|`--l1.chain_id`|Chain id of L1 chain endpoint to use|`1`|| -|`--l1.epoch-poll-interval`|Poll interval for retrieving new L1 epoch updates such as safe and finalized block changes. Disabled if 0 or negative.|`6m24s`|| -|`--l1.min-duration-blobs-request`|Min duration for blobs sidecars request|`1572864`|| -|`--l1.rpc`|Address of L1 User JSON-RPC endpoint to use (eth namespace required)||✓| -|`--l2.chain_id`|Chain id of L2 chain endpoint to use|`3333`|| -|`--log.color`|Color the log output if in terminal mode||| -|`--log.format`|Format the log output. Supported formats: 'text', 'terminal', 'logfmt', 'json', 'json-pretty',|`text`|| -|`--log.level`|The lowest log level that will be output|`info`|| -|`--metrics.enable`|Enable metrics||| -|`--miner.enabled`|Storage mining enabled||| -|`--miner.gas-price`|Gas price for mining transactions||| -|`--miner.min-profit`|Minimum profit for mining transactions|`0`|| -|`--miner.priority-gas-price`|Priority gas price for mining transactions||| -|`--miner.threads-per-shard`|Number of threads per shard|`runtime.NumCPU() x 2`|| -|`--miner.zk-working-dir`|Path to the snarkjs folder|`build/bin`|| -|`--miner.zk-prover-mode`|ZK prover mode, 1: one proof per sample, 2: one proof for multiple samples|`2`|| -|`--miner.zkey`|zkey file name which should be put in the snarkjs folder|`blob_poseidon2.zkey`|| -|`--network`|Predefined L1 network selection. Available networks: devnet||| -|`--p2p.advertise.ip`|The IP address to advertise in Discv5, put into the ENR of the node. This may also be a hostname / domain name to resolve to an IP.||| -|`--p2p.advertise.tcp`|The TCP port to advertise in Discv5, put into the ENR of the node. Set to p2p.listen.tcp value if 0.|`0`|| -|`--p2p.advertise.udp`|The UDP port to advertise in Discv5 as fallback if not determined by Discv5, put into the ENR of the node. Set to p2p.listen.udp value if 0.|`0`|| -|`--p2p.ban.peers`|Enables peer banning. This should ONLY be enabled once certain peer scoring is working correctly.||| -|`--p2p.bootnodes`|Comma-separated base64-format ENR list. Bootnodes to start discovering other node records from.||| -|`--p2p.disable`|Completely disable the P2P stack, if P2P is disabled, mining will not start automatically.||| -|`--p2p.discovery.path`|Discovered ENRs are persisted in a database to recover from a restart without having to bootstrap the discovery process again. Set to 'memory' to never persist the peerstore.|`esnode_discovery_db`|| -|`--p2p.listen.ip`|IP to bind LibP2P and Discv5 to|`0.0.0.0`|| -|`--p2p.listen.tcp`|TCP port to bind LibP2P to. Any available system port if set to 0.|`9222`|| -|`--p2p.listen.udp`|UDP port to bind Discv5 to. Same as TCP port if left 0.|`0`|| -|`--p2p.nat`|Enable NAT traversal with PMP/UPNP devices to learn external IP.||| -|`--p2p.max.request.size`|max request size is the maximum number of bytes to request from a remote peer. It is value should not larger than 8 * 1024 * 1024. if you have good network condition, you can increase the max request size to improve the sync performance.|1048576|| -|`--p2p.sync.concurrency`|sync concurrency is the number of chunks to split a shard into to allow concurrent retrievals.|16|| -|`--p2p.fill-empty.concurrency`|fill empty concurrency is the number of threads to concurrently fill encoded empty blobs.|NumCPU - 2|| -|`--p2p.no-discovery`|Disable Discv5 (node discovery)||| -|`--p2p.peers.grace`|Grace period to keep a newly connected peer around, if it is not misbehaving.|`30s`|| -|`--p2p.peers.hi`|High-tide peer count. The node starts pruning peer connections slowly after reaching this number.|`30`|| -|`--p2p.peers.lo`|Low-tide peer count. The node actively searches for new peer connections if below this amount.|`20`|| -|`--p2p.peerstore.path`|Peerstore database location. Persisted peerstores help recover peers after restarts. Set to 'memory' to never persist the peerstore. Peerstore records will be pruned / expire as necessary. Warning: a copy of the priv network key of the local peer will be persisted here.|`esnode_peerstore_db`|| -|`--p2p.priv.path`|Read the hex-encoded 32-byte private key for the peer ID from this txt file. Created if not already exists.Important to persist to keep the same network identity after restarting, maintaining the previous advertised identity.|`esnode_p2p_priv.txt`|| -|`--p2p.score.bands`|Sets the peer score bands used primarily for peer score metrics. Should be provided in following format: \:\