From 76811f0354ffd26d908433bcfce97e0ae99134f4 Mon Sep 17 00:00:00 2001 From: Georgi Lazarov Date: Fri, 8 Nov 2024 21:55:59 +0200 Subject: [PATCH] feat: Improve README.md structure (#323) Signed-off-by: georgi-l95 Signed-off-by: Atanas Atanasov Co-authored-by: Atanas Atanasov --- README.md | 92 ++++++++++--------- docs/Protocol/README.md | 2 +- docs/Protocol/publishBlockStream.md | 9 ++ docs/overview.md | 23 +++++ docs/release.md | 75 ++++++++------- server/README.md | 39 ++++++++ server/docs/configuration.md | 18 ++++ .../bidi-producer-consumers-streaming.md | 51 ++++++---- server/docs/design/block-persistence.md | 18 +++- server/docs/metrics.md | 38 +++++--- server/docs/quickstart.md | 66 +++++++++++++ simulator/README.md | 84 +++-------------- simulator/docs/configuration.md | 39 ++++++++ simulator/docs/quickstart.md | 60 ++++++++++++ tools/README.md | 52 +++++++---- tools/docs/quickstart.md | 41 +++++++++ 16 files changed, 509 insertions(+), 198 deletions(-) create mode 100644 docs/overview.md create mode 100644 server/README.md create mode 100644 server/docs/configuration.md create mode 100644 server/docs/quickstart.md create mode 100644 simulator/docs/configuration.md create mode 100644 simulator/docs/quickstart.md create mode 100644 tools/docs/quickstart.md diff --git a/README.md b/README.md index aaa3e7b18..4c523951c 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,48 @@ -# hedera-block-node -The block node is a new and unique node designed to consume the blocks streams, maintain state and expose additional targeted value adding APIs to the Hedera community. -More details to come as this is work in progress +# Hedera Block Node + +Implementation of the Hedera Block Node, which is responsible for consuming the block streams, maintaining state and exposing additional targeted value adding APIs to the Hedera community. + +## Table of Contents + +1. [Project Links](#project-links) +1. [Prerequisites](#prerequisites) +1. [Overview of child modules](#overview-of-child-modules) +1. [Getting Started](#getting-started) +1. [Support](#support) +1. [Contributing](#contributing) +1. [Code of Conduct](#code-of-conduct) +1. [Security](#-security) +1. [License](#license) + +## Project Links + +[![Build Application](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml/badge.svg?branch=main)](https://github.com/hashgraph/hedera-block-node/actions/workflows/build-application.yaml) +[![E2E Test Suites](https://github.com/hashgraph/hedera-block-node/actions/workflows/e2e-tests.yaml/badge.svg?branch=main)](https://github.com/hashgraph/hedera-block-node/actions/workflows/e2e-tests.yaml) +[![codecov](https://codecov.io/github/hashgraph/hedera-block-node/graph/badge.svg?token=OF6T6E8V7U)](https://codecov.io/github/hashgraph/hedera-block-node) + +[![Latest Version](https://img.shields.io/github/v/tag/hashgraph/hedera-block-node?sort=semver&label=version)](README.md) +[![Made With](https://img.shields.io/badge/made_with-java-blue)](https://github.com/hashgraph/hedera-block-node/) +[![Development Branch](https://img.shields.io/badge/docs-quickstart-green.svg)](docs/overview.md) +[![License](https://img.shields.io/badge/license-apache2-blue.svg)](LICENSE) + +## Prerequisites + +- Java 21 (temurin recommended) +- Gradle (using the wrapper `./gradlew` is highly recommended) +- Docker (recommended for running the projects) +- IntelliJ IDEA (recommended for development) + +## Overview of child modules + +- `server`: implementation of the block node, which contains the main application and all the necessary code to run the block node. +- `simulator`: A simulator for the block node, which can be used to test the block node in a local environment. +- `common`: Module responsible for holding common literals, utilities and types used by the other modules. +- `suites`: A set of e2e tests that can be used to verify the correctness of the block node. +- `tools`: A set of command line tools for working with block stream files. + +## Getting Started + +Refer to the [Hedera Block Node Documentation Overview](docs/overview.md) for more information about the project, design and guides. ## Support @@ -20,46 +62,10 @@ This project is governed by the participating, you are expected to uphold this code of conduct. Please report unacceptable behavior to [oss@hedera.com](mailto:oss@hedera.com). -## License - -[Apache License 2.0](LICENSE) - -# 🔐 Security - -Please do not file a public ticket mentioning the vulnerability. Refer to the security policy defined in the [SECURITY.md](https://github.com/hashgraph/hedera-sourcify/blob/main/SECURITY.md). - ---- - -# Usage - -## Configuration +## 🔐 Security -| Environment Variable | Description | Default Value | -|-----------------------------------|---------------------------------------------------------------------------------------------------------------|---------------| -| PERSISTENCE_STORAGE_ROOT_PATH | The root path for the storage, if not provided will attempt to create a `data` on the working dir of the app. | | -| CONSUMER_TIMEOUT_THRESHOLD_MILLIS | Time to wait for subscribers before disconnecting in milliseconds | 1500 | -| SERVICE_DELAY_MILLIS | Service shutdown delay in milliseconds | 500 | -| MEDIATOR_RING_BUFFER_SIZE | Size of the ring buffer used by the mediator (must be a power of 2) | 67108864 | -| NOTIFIER_RING_BUFFER_SIZE | Size of the ring buffer used by the notifier (must be a power of 2) | 2048 | - -# Starting locally: -```bash -./gradlew run -``` - -In debug mode, you can attach a debugger to the port 5005. -```bash -./gradlew run --debug-jvm -``` - -Also you can run on docker locally: -```bash -./gradlew startDockerContainer -``` - -# Running Tests -1) ./gradlew build +Please do not file a public ticket mentioning the vulnerability. Refer to the security policy defined in the [SECURITY.md](https://github.com/hashgraph/hedera-block-node/blob/main/SECURITY.md). +## License -### Code Coverage -[![codecov](https://codecov.io/github/hashgraph/hedera-block-node/graph/badge.svg?token=OF6T6E8V7U)](https://codecov.io/github/hashgraph/hedera-block-node) +[Apache License 2.0](LICENSE) diff --git a/docs/Protocol/README.md b/docs/Protocol/README.md index 8dc715626..3b7191fdb 100644 --- a/docs/Protocol/README.md +++ b/docs/Protocol/README.md @@ -8,4 +8,4 @@ behavior of both sides of that API call, including common error conditions. ## Contents | Document | API call | Description | |:---|---:|:---| -| publishBlockStream.md | `publishBlockStream` | The communication between a publisher and a block node when publishing a block stream from an authoritative source such as a consensus node.| +| [publishBlockStream.md](publishBlockStream.md) | `publishBlockStream` | The communication between a publisher and a block node when publishing a block stream from an authoritative source such as a consensus node. | diff --git a/docs/Protocol/publishBlockStream.md b/docs/Protocol/publishBlockStream.md index c1ddefa28..0d607390d 100644 --- a/docs/Protocol/publishBlockStream.md +++ b/docs/Protocol/publishBlockStream.md @@ -1,5 +1,14 @@ # Block Node Connect Protocol for `publishBlockStream` +## Table of Contents +1. [Abstract](#abstract) + 1. [Reliability note](#reliability-note) +1. [Definitions](#definitions) +1. [Base Protocol](#base-protocol) + 1. [Base Protocol Diagram](#base-protocol-diagram) +1. [Error Handling](#error-handling) + 1. [Error Handling Diagram](#error-handling-diagram) + ## Abstract This protocol describes how a Publisher and Block Node SHALL interact for the `publishBlockStream` API. The protocol provides for either side of the diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 000000000..018e76e01 --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,23 @@ +# Hedera Block Node Documentation Overview + +This document will outline the structure and content of Block Node documentation. + +## Table of Contents + +1. [Projects](#projects) +1. [Concepts](#concepts) +1. [Release](#release) + +## Projects + +- [Hedera Block Node](../server/README.md): Provides an introduction to Hedera Block Node, its purpose and how it can be used. +- [Simulator](../simulator/README.md): Offers instructions on how the Simulator can be utilized. +- [Tools](../tools/README.md): Provides information on the command line tools available. + +## Concepts + +- [Hedera Block Node Protocol](Protocol/README.md): Overview of the Hedera Block Node Protocol + +## Release + +- [Release Process Documentation](release.md): Details the steps involved in creating a new release, tagging, and publishing artifacts. diff --git a/docs/release.md b/docs/release.md index 55504a089..ce86b0900 100644 --- a/docs/release.md +++ b/docs/release.md @@ -1,57 +1,68 @@ -## Release Process Documentation +# Release Process Documentation -### Overview +## Table of Contents + +1. [Overview](#overview) +1. [Kickstart Release Process](#kickstart-release-process) +1. [Release Automation Workflow](#release-automation-workflow) + 1. [Steps](#steps) +1. [Publish Release Workflow](#publish-release-workflow) +1. [Mermaid Diagram](#mermaid-diagram) +1. [Release Meta Process](#release-meta-process) +1. [Conclusion](#conclusion) + +## Overview This document outlines the release process for our project, detailing the steps involved in creating a new release, tagging, and publishing artifacts. -### Kickstart Release Process +## Kickstart Release Process 1. **Milestone and Label Check**: - - Ensure all Pull Requests (PRs) have a milestone set and appropriate labels. This is important for generating accurate release notes. - - There is a CI check to ensure that all PRs have a milestone and labels set. If not, the PR will not be merged. - - Make sure all PRs and Issues related to the related milestone are already closed, if not closed them before proceeding or move them to the next milestone. + - Ensure all Pull Requests (PRs) have a milestone set and appropriate labels. This is important for generating accurate release notes. + - There is a CI check to ensure that all PRs have a milestone and labels set. If not, the PR will not be merged. + - Make sure all PRs and Issues related to the related milestone are already closed, if not closed them before proceeding or move them to the next milestone. 2. **Trigger Release Automation Workflow**: - - Manually trigger the `release-automation.yaml` workflow with the desired version to be released. - - The workflow will create a new release branch, bump the version, and tag the release. - - The release branch will be used for any patch versions or hot-fixes. - - If the release branch already exists, the workflow should be triggered using the branch release, and the version should be bumped accordingly. + - Manually trigger the `release-automation.yaml` workflow with the desired version to be released. + - The workflow will create a new release branch, bump the version, and tag the release. + - The release branch will be used for any patch versions or hot-fixes. + - If the release branch already exists, the workflow should be triggered using the branch release, and the version should be bumped accordingly. - for example: if `release/0.1` exists, and you want to release `0.1.1`, trigger the workflow with `0.1.1` as the semver input and select `release/0.1` on "Use workflow from" dropdown. -### Release Automation Workflow +## Release Automation Workflow The release process is automated using a GitHub Actions workflow (`release-automation.yaml`). The workflow must be triggered manually with a SemVer version input, which specifies the version to be released. The input can be: - `x.n.0-rc1` for release candidates - `x.n.0` for general availability (GA) versions - `x.n.p` for patch versions, including hot-fixes -#### Steps: +### Steps: 1. **Create or Update Release Branch**: - - Create a new release branch `release/0.n` if it doesn't exist. - - If the release branch was created, means it's a new release, create a PR on the main branch to increase the snapshot version to the next version. - - Bump the version to the input provided in the release branch. + - Create a new release branch `release/0.n` if it doesn't exist. + - If the release branch was created, means it's a new release, create a PR on the main branch to increase the snapshot version to the next version. + - Bump the version to the input provided in the release branch. 2. **Tagging**: - - After bumping the version on the release branch, tag the version as `vX.n.p` (where `X.n.p` is the input). - - Push the tag, triggering the `Publish Release Workflow`. + - After bumping the version on the release branch, tag the version as `vX.n.p` (where `X.n.p` is the input). + - Push the tag, triggering the `Publish Release Workflow`. 3. **Release Notes and Milestone**: - - Create release notes for the new Tag. - - Close the milestone associated with the release. - - Upload distribution artifacts to the release. - - Publish the artifacts to Sonatype Nexus / Maven Central. + - Create release notes for the new Tag. + - Close the milestone associated with the release. + - Upload distribution artifacts to the release. + - Publish the artifacts to Sonatype Nexus / Maven Central. -### Publish Release Workflow +## Publish Release Workflow The `Publish Release Workflow` (`publish-release.yaml`) is triggered by the Tag pushed in the previous step. This workflow performs the following tasks: 1. **Docker Image Creation and Publishing**: - - Create a Docker image with the new release version and publish it to the GitHub Container Registry (`ghcr.io`). + - Create a Docker image with the new release version and publish it to the GitHub Container Registry (`ghcr.io`). 2. **Helm Chart Publishing**: - - Publish a new Helm chart with the same version to the custom GitHub Pages of the repo in the `Charts` folder. + - Publish a new Helm chart with the same version to the custom GitHub Pages of the repo in the `Charts` folder. -### Mermaid Diagram +## Mermaid Diagram ```mermaid graph TD @@ -84,20 +95,20 @@ graph TD end ``` -### Release Meta Process +## Release Meta Process The meta process typically follows these steps: 1. **Initial Release Candidate (RC)**: - - Trigger the Release Automation Workflow with an `rc` version (e.g., `x.n.0-rcY`). - - Perform integration and performance testing on this version. + - Trigger the Release Automation Workflow with an `rc` version (e.g., `x.n.0-rcY`). + - Perform integration and performance testing on this version. 2. **General Availability (GA)**: - - Once testing is successful, trigger the Release Automation Workflow again for the same version but as a GA version (e.g., `x.n.0`). + - Once testing is successful, trigger the Release Automation Workflow again for the same version but as a GA version (e.g., `x.n.0`). 3. **Patch Versions**: - - For any patch versions, changes will be merged (cherry-picked from main) into the release branch. - - Start a new release automation process for the patch version (e.g., `x.n.p`). + - For any patch versions, changes will be merged (cherry-picked from main) into the release branch. + - Start a new release automation process for the patch version (e.g., `x.n.p`). ```mermaid graph TD @@ -110,10 +121,8 @@ graph TD D1 --> E1[Merge Patches from Main to Release Branch] E1 --> F1[Trigger Release Automation Workflow for x.n.p] F1 --> G1[Patch Version Released] - ``` ## Conclusion This release process ensures that all steps are automated, from checking PR milestones and labels to creating branches, tagging releases, and publishing Docker images and Helm charts. By following this documentation, you can streamline the release workflow and maintain consistency across all releases. - diff --git a/server/README.md b/server/README.md new file mode 100644 index 000000000..7127e5c8b --- /dev/null +++ b/server/README.md @@ -0,0 +1,39 @@ +# Block Node Application + +## Table of Contents + +1. [Overview](#overview) +1. [Prerequisites](#prerequisites) +1. [Configuration](#configuration) +1. [Metrics](#metrics) +1. [Design](#design) + 1. [Block Persistence](#block-persistence) + 1. [Bi-directional Producer/Consumer Streaming with gRPC](#bi-directional-producerconsumer-streaming-with-grpc) + +## Overview + +The Block Stream Application is designed handle the output of a Hedera Node, which would be in form of Block Stream. +By handling we can understand verifying, storing and applying needed state changes. +It uses various configuration sources and dependency injection to manage its components. + +## Configuration + +Refer to the [Configuration](docs/configuration.md) for configuration options. + +## Quickstart + +Refer to the [Quickstart](docs/quickstart.md) for a quick guide on how to get started with the application. + +## Metrics + +Refer to the [Metrics](docs/metrics.md) for metrics available in the system. + +## Design + +### Block Persistence + +Refer to the [Block Persistence](docs/design/block-persistence.md) for details on how blocks are persisted. + +### Bi-directional Producer/Consumer Streaming with gRPC + +Refer to the [Bi-directional Producer/Consumer Streaming with gRPC](docs/design/bidi-producer-consumers-streaming.md) for details on how the gRPC streaming is implemented. diff --git a/server/docs/configuration.md b/server/docs/configuration.md new file mode 100644 index 000000000..50f2cd6d9 --- /dev/null +++ b/server/docs/configuration.md @@ -0,0 +1,18 @@ +# Server Configuration + +The components of the Hedera Block Node all support loading configuration via the +environment. + +## Default Values + +The default configuration allows users to quickly get up and running without having to configure anything. This provides +ease of use at the trade-off of some insecure default configuration. Most configuration settings have appropriate +defaults and can be left unchanged. It is recommended to browse the properties below and adjust to your needs. + +| Environment Variable | Description | Default Value | +|:---|:---|---:| +| PERSISTENCE_STORAGE_ROOT_PATH | The root path for the storage, if not provided will attempt to create a `data` on the working dir of the app. | | +| CONSUMER_TIMEOUT_THRESHOLD_MILLIS | Time to wait for subscribers before disconnecting in milliseconds | 1500 | +| SERVICE_DELAY_MILLIS | Service shutdown delay in milliseconds | 500 | +| MEDIATOR_RING_BUFFER_SIZE | Size of the ring buffer used by the mediator (must be a power of 2) | 67108864 | +| NOTIFIER_RING_BUFFER_SIZE | Size of the ring buffer used by the notifier (must be a power of 2) | 2048 | diff --git a/server/docs/design/bidi-producer-consumers-streaming.md b/server/docs/design/bidi-producer-consumers-streaming.md index 08215d405..326512194 100644 --- a/server/docs/design/bidi-producer-consumers-streaming.md +++ b/server/docs/design/bidi-producer-consumers-streaming.md @@ -1,5 +1,29 @@ # Bi-directional Producer/Consumer Streaming with gRPC +## Table of Contents + +1. [Purpose](#purpose) +1. [Goals](#goals) +1. [Terms](#terms) +1. [Block Node gRPC Streaming Services API](#block-node-grpc-streaming-services-api) +1. [Approaches](#approaches) + 1. [Approach 1: Directly passing BlockItems from `ProducerBlockItemObserver` to N `ConsumerBlockItemObserver`s](#approach-1-directly-passing-blockitems-from-producerblockitemobserver-to-n-consumerblockitemobservers) + 1. [Approach 2: Use a shared data structure between `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s. Consumers busy-wait for new BlockItems](#approach-2-use-a-shared-data-structure-between-producerblockitemobserver-and-consumerblockitemobservers-consumers-busy-wait-for-new-blockitems) + 1. [Approach 3: Use a shared data structure between `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s. Use downstream consumer BlockItemResponses to drive the process of sending new BlockItems](#approach-3-use-a-shared-data-structure-between-producerblockitemobserver-and-consumerblockitemobservers-use-downstream-consumer-blockitemresponses-to-drive-the-process-of-sending-new-blockitems) + 1. [Approach 4: Shared data structure between producer and consumer services. Leveraging the LMAX Disruptor library to manage inter-process pub/sub message-passing between producer and consumers via RingBuffer](#approach-4-shared-data-structure-between-producer-and-consumer-services-leveraging-the-lmax-disruptor-library-to-manage-inter-process-pubsub-message-passing-between-producer-and-consumers-via-ringbuffer) +1. [Design](#design) + 1. [Producer Registration Flow](#producer-registration-flow) + 1. [Consumer Registration Flow](#consumer-registration-flow) + 1. [Runtime Streaming](#runtime-streaming) + 1. [Entities](#entities) +1. [Diagrams](#diagrams) + 1. [Producer Registration Flow](#producer-registration-flow-1) + 1. [Consumer Registration Flow](#consumer-registration-flow-1) + 1. [Class Diagram of all Entities and their Relationships](#class-diagram-of-all-entities-and-their-relationships) + 1. [Runtime Stream of BlockItems from Producer to Consumers](#runtime-stream-of-blockitems-from-producer-to-consumers) + +--- + ## Purpose A primary use case of the `hedera-block-node` is to stream live BlockItems (see Terms section) from a producer @@ -17,15 +41,15 @@ point for custom logic is an implementation of `GrpcService`. ## Goals 1) Consumers must be able to dynamically subscribe and unsubscribe from the live stream of BlockItems emitted by the - producer. When a consumer subscribes to the stream, they will begin receiving BlockItems at the start of the next Block. - BlockItems transiting before the start of the next Block will not be sent to that downstream consumer. + producer. When a consumer subscribes to the stream, they will begin receiving BlockItems at the start of the next Block. + BlockItems transiting before the start of the next Block will not be sent to that downstream consumer. 2) Correct, in-order streaming delivery of BlockItems from a producer to all registered consumers. 3) Minimize latency between the producer and consumers. 4) Minimize CPU resources consumed by the producer and consumers. --- -### Terms +## Terms **BlockItem** - The BlockItem is the primary data structure passed between the producer, the `hedera-block-node` and consumers. A defined sequence of BlockItems represent a Block when stored on the `hedera-block-node`. @@ -51,14 +75,13 @@ unsubscribed from a producer so that internal objects can be cleaned up and reso --- -### Block Node gRPC Streaming Services API +## Block Node gRPC Streaming Services API The Block Node gRPC Streaming Services API is now aligned with the names and simplified types defined in the [`hedera-protobufs` repository on the `continue-block-node` branch](https://github.com/hashgraph/hedera-protobufs/blob/25783427575ded59d06d6bf1ed253fd24ef3c437/block/block_service.proto#L701-L742). --- - ## Approaches: All the following approaches require integrating with Helidon 4.x.x gRPC services to implement the bidirectional @@ -80,7 +103,7 @@ BlockItem to the downstream consumer. Helidon invokes `ConsumerBlockItemObserver from the consumer in receipt of BlockItems. -### Approach 1: Directly passing BlockItems from `ProducerBlockItemObserver` to N `ConsumerBlockItemObserver`s. +### Approach 1: Directly passing BlockItems from `ProducerBlockItemObserver` to N `ConsumerBlockItemObserver`s. Directly passing BlockItems from the `ProducerBlockItemObserver` to N `ConsumerBlockItemObserver`s without storing BlockItems in an intermediate data structure. This approach was the basis for one of the first implementations of gRPC @@ -89,8 +112,8 @@ Live Streaming (see [BlockNode Issue 21](https://github.com/hashgraph/hedera-blo Drawbacks: 1) Each `ProducerBlockItemObserver` must iterate over the list of subscribed consumers to pass the BlockItem to each `ConsumerBlockItemObserver` before saving the BlockItem to disk and issuing a BlockItemResponse back to the producer. - The linear scaling of consumers will aggregate latency resulting in the last consumer in the list to be penalized - with the sum of the latencies of all consumers before it. + The linear scaling of consumers will aggregate latency resulting in the last consumer in the list to be penalized + with the sum of the latencies of all consumers before it. 2) Dynamically subscribing/unsubscribing `ConsumerBlockItemObserver`s while deterministically broadcasting BlockItems to each consumer in the correct order complicates and slows down the process. It requires thread-safe data structures and synchronization on all reads and writes to ensure new/removed subscribers do not disrupt the @@ -117,7 +140,6 @@ Drawbacks: downstream consumer in a timely way. The aggressive behavior of the busy-wait could complicate future use cases requiring downstream consumer response processing. - ### Approach 3: Use a shared data structure between `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s. Use downstream consumer BlockItemResponses to drive the process of sending new BlockItems. With this approach, the `ProducerBlockItemObserver` will store BlockItems in a shared data structure before immediately @@ -159,7 +181,7 @@ Advantages: Drawbacks: 1) The Disruptor library is a third-party library requiring ramp-up time and integration effort to use it correctly and - effectively. + effectively. 2) Leveraging the Disruptor library requires the communication between the `ProducerBlockItemObserver` and `ConsumerBlockItemObserver`s to be affiliated by subscribing/unsubscribing the downstream consumers to receive the latest BlockItems from the producer via the Disruptor RingBuffer. The process of managing these subscriptions to @@ -193,7 +215,6 @@ the `ResponseStreamObserver` managed by Helidon for transmitting BlockItemRespon `BlockItemStreamService` will also subscribe the `ConsumerBlockItemObserver` to the `StreamMediator` to receive the streaming BlockItems from the producer. - ### Runtime Streaming At runtime, the `ProducerBlockItemObserver` will receive the latest BlockItem from the producer via Helidon and will @@ -250,20 +271,18 @@ the latest BlockItem to the ConsumerBlockItemObserver when it is available in th **BlockPersistenceHandler** - The BlockPersistenceHandler is responsible for writing the latest BlockItem to disk. --- -## Diagrams +## Diagrams ### Producer Registration Flow ![Producer Registration](assets/00036-producer-registration.png) - ### Consumer Registration Flow ![Consumer Registration](assets/00036-consumer-registration.png) - -### Class Diagram of all Entities and their Relationships +### Class Diagram of all Entities and their Relationships ![Class Diagram](assets/00036-demo-disruptor-class-diagram.png) @@ -271,6 +290,4 @@ the latest BlockItem to the ConsumerBlockItemObserver when it is available in th ![Sequence Diagram](assets/00036-refactor-demo-disruptor.png) - --- - diff --git a/server/docs/design/block-persistence.md b/server/docs/design/block-persistence.md index ea10f49ac..561c9b428 100644 --- a/server/docs/design/block-persistence.md +++ b/server/docs/design/block-persistence.md @@ -1,5 +1,15 @@ # Block Persistence +## Table of Contents + +1. [Purpose](#purpose) +1. [Goals](#goals) +1. [Terms](#terms) +1. [Entities](#entities) +1. [Design](#design) + +--- + ## Purpose The main objective of the `hedera-block-node` project is to replace the storage of Consensus Node artifacts (e.g. @@ -9,7 +19,7 @@ when they arise. --- -### Goals +## Goals 1) BlockItems streamed from a producer (e.g. Consensus Node) must be collated and persisted as a Block. Per the specification, a Block is an ordered list of BlockItems. How the Block is persisted is an implementation detail. @@ -17,7 +27,7 @@ when they arise. --- -### Terms +## Terms **BlockItem** - A BlockItem is the primary data structure passed between the producer, the `hedera-block-node` and consumers. The BlockItem description and protobuf definition are maintained in the `hedera-protobuf` @@ -29,7 +39,7 @@ BlockItems. The Block description and protobuf definition are maintained in the --- -### Entities +## Entities **BlockReader** - An interface defining methods used to read a Block from storage. It represents a lower-level component whose implementation is directly responsible for reading a Block from storage. @@ -42,7 +52,7 @@ component whose implementation is directly responsible for removing a Block from --- -### Design +## Design The design for `Block` persistence is fairly straightforward. Block server objects should use the persistence entity interfaces to read, write and remove `Block`s from storage. `BlockItem`s streamed from a producer are read off the wire diff --git a/server/docs/metrics.md b/server/docs/metrics.md index 37c7277d8..41021a680 100644 --- a/server/docs/metrics.md +++ b/server/docs/metrics.md @@ -1,9 +1,19 @@ # Metrics This document describes the metrics that are available in the system, its purpose, and how to use them. +## Table of Contents + +1. [Overview](#overview) +1. [Purpose](#purpose) +1. [Configuration](#configuration) + 1. [Prometheus](#prometheus) +1. [Usage](#usage) + 1. [Local Development](#local-development) +1. [Existing Metrics](#existing-metrics) + ## Overview -We are using Prometheus to collect metrics from the system, through the use of [Hedera Platform SDK - Metrics](https://github.com/hashgraph/hedera-services/blob/develop/platform-sdk/docs/base/metrics/metrics.md). +We are using Prometheus to collect metrics from the system, through the use of [Hedera Platform SDK - Metrics](https://github.com/hashgraph/hedera-services/blob/develop/platform-sdk/docs/base/metrics/metrics.md). ## Purpose @@ -15,13 +25,14 @@ The purpose of metrics is to provide a way to measure the performance of the sys Prefix: prometheus, ie. `prometheus.configKey` -| ConfigKey | Description | Default | -|----------------------------|---------------------------------------------------------------------------------------|-------------| -| enableEndpoint | either `true` or `false`. Enables or disables the endpoint for metrics | true | -| endpointPortNumber | Port of the Prometheus endpoint | 9999 | -| endpointMaxBacklogAllowed | The maximum number of incoming TCP connections which the system will queue internally | 1 | +| ConfigKey | Description | Default | +|:---|:---|---:| +| enableEndpoint | either `true` or `false`. Enables or disables the endpoint for metrics | true | +| endpointPortNumber | Port of the Prometheus endpoint | 9999 | +| endpointMaxBacklogAllowed | The maximum number of incoming TCP connections which the system will queue internally | 1 | ## Usage + All classes that need to observe metrics can get them from the BlockNodeContext. If a new metric is needed, it can be added to MetricsService, and then used in the classes that need it. This is to avoid having to define the metrics in several places. @@ -30,6 +41,7 @@ MetricsService centralizes the creation of metrics and provides a way to access To check the metrics you can access the Prometheus endpoint at `http://localhost:9999/metrics`. ### Local Development + For developers, when using the gradle task `startDockerContainer` it will automatically start a prometheus and grafana services preconfigured locally with credentials: username `admin` and password `admin` and the dashboard already provisioned with the current metrics and widgets. Dashboard is called `Block-Node Server Dashboard` and its source is kept on folder: `server/docker/metrics/dashboards` as: `block-node-server.json`. @@ -38,15 +50,13 @@ When doing changes to the dashboard on grafana is important to copy the json to If needed to create another dashboard is possible to include it by adding it to the same folder. - ## Existing Metrics All metrics have `hedera_block_node` prefix. -| Metric Name | Description | Type | -|---------------------------------|---------------------------------------|-----------| -| live_block_items | The number of block items received | Counter | -| blocks_persisted | the number of blocks persisted | Counter | -| subscribers | The number of subscribers | Gauge | -| single_blocks_retrieved | the number of single blocks requested | Counter | - +| Metric Name | Description | Type | +|:---|:---|---:| +| live_block_items | The number of block items received | Counter | +| blocks_persisted | the number of blocks persisted | Counter | +| subscribers | The number of subscribers | Gauge | +| single_blocks_retrieved | the number of single blocks requested | Counter | diff --git a/server/docs/quickstart.md b/server/docs/quickstart.md new file mode 100644 index 000000000..4682a47a4 --- /dev/null +++ b/server/docs/quickstart.md @@ -0,0 +1,66 @@ +# Quickstart of the Server + +## Table of Contents + +1. [Configuration](#configuration) +1. [Running locally](#running-locally) + 1. [Build the Server](#build-the-server) + 1. [Run the Server](#run-the-server) + 1. [Run the Server with Debug](#run-the-server-with-debug) + 1. [Stop the Server](#stop-the-server) + +## Configuration + +Refer to the [Configuration](configuration.md) for configuration options. + +## Running locally: + +- Server subproject qualifier: `:server` +- Assuming your working directory is the repo root + +> **NOTE:** one may use the `-p` flag for `./gradlew` in order to avoid +> specifying the target subproject repeatedly on each task when running +> multiple tasks. When running only a single task, however, it is +> recommended to use the project qualifier (i.e. `:server:`) for +> both simplicity and clarity. + +### Build the Server + +> **NOTE:** if you have not done so already, it is +> generally recommended to build the entire repo first: +> ```bash +> ./gradlew clean build -x test +> ``` + +1. To quickly build the Server sources (without running tests), do the following: + ```bash + ./gradlew -p server clean build -x test + ``` + +1. To build the Server docker image, do the following: + ```bash + ./gradlew :server:createDockerImage + ``` + +### Run the Server + +1. To start the Server, do the following: + ```bash + ./gradlew :server:startDockerContainer + ``` + +### Run the Server with Debug + +1. To start the Server with debug enabled, do the following: + ```bash + ./gradlew :server:startDockerDebugContainer + ``` + +1. Attach your remote jvm debugger to port 5005. + +### Stop the Server + +1. To stop the Server do the following: + ```bash + ./gradlew :server:stopDockerContainer + ``` diff --git a/simulator/README.md b/simulator/README.md index 837871a15..e9c34c577 100644 --- a/simulator/README.md +++ b/simulator/README.md @@ -1,93 +1,39 @@ # Block Stream Simulator +## Table of Contents +1. [Overview](#overview) +1. [Project Design Structure](#project-design-structure) +1. [Configuration](#configuration) +1. [Quickstart](#quickstart) + ## Overview The Block Stream Simulator is designed to simulate block streaming for Hedera Hashgraph. It uses various configuration sources and dependency injection to manage its components. -## Prerequisites - -- Java 21 -- Gradle -- IntelliJ IDEA (recommended for development) - ## Project Design Structure Uses Dagger2 for dependency injection, the project has a modular structure and divides the Dagger dependencies into modules, but all modules used can be found at the root Injection Module: ```plaintext src/java/com/hedera/block/simulator/BlockStreamSimulatorInjectionModule.java ``` + Entry point for the project is `BlockStreamSimulator.java`, in wich the main method is located and has 2 functions: 1. Create/Load the Application Configuration, it does this using Hedera Platform Configuration API. -2. Create a DaggerComponent and instantiate the BlockStreamSimulatorApp class using the DaggerComponent and it registered dependencies. -3. Start the BlockStreamSimulatorApp, contains the orchestration of the different parts of the simulation using generic interfaces and handles the rate of streaming and the exit conditions. +1. Create a DaggerComponent and instantiate the BlockStreamSimulatorApp class using the DaggerComponent and it registered dependencies. +1. Start the BlockStreamSimulatorApp, contains the orchestration of the different parts of the simulation using generic interfaces and handles the rate of streaming and the exit conditions. The BlockStreamSimulatorApp consumes other services that are injected using DaggerComponent, these are: 1. **generator:** responsible for generating blocks, exposes a single interface `BlockStreamManager` and several implementations 1. BlockAsDirBlockStreamManager: generates blocks from a directory, each folder is a block, and block-items are single 'blk' or 'blk.gz' files. - 2. BlockAsFileBlockStreamManager: generates blocks from a single file, each file is a block, used to the format of the CN recordings. (since it loads blocks on memory it can stream really fast, really useful for simple streaming tests) - 3. BlockAsFileLargeDataSets: similar to BlockAsFileBLockStreamManager, but designed to work with GB folders with thousands of big blocks (since it has a high size block and volume of blocks, is useful for performace, load and stress testing) -2. **grpc:** responsible for the communication with the Block-Node, currently only has 1 interface `PublishStreamGrpcClient` and 1 Implementation, however also exposes a `PublishStreamObserver' + 1. BlockAsFileBlockStreamManager: generates blocks from a single file, each file is a block, used to the format of the CN recordings. (since it loads blocks on memory it can stream really fast, really useful for simple streaming tests) + 1. BlockAsFileLargeDataSets: similar to BlockAsFileBLockStreamManager, but designed to work with GB folders with thousands of big blocks (since it has a high size block and volume of blocks, is useful for performace, load and stress testing) +1. **grpc:** responsible for the communication with the Block-Node, currently only has 1 interface `PublishStreamGrpcClient` and 1 Implementation, however also exposes a `PublishStreamObserver' ## Configuration -There are 2 configuration sets: -1. BlockStreamConfig: contains the configuration for the Block Stream Simulator logic and the generation module. -2. GrpcConfig: contains the configuration for the gRPC communication with the Block-Node. - -### BlockStreamConfig -Uses the prefix `blockStream` so all properties should start with `blockStream.` - -| Key | Description | Default Value | -|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------| -| `delayBetweenBlockItems` | The delay between each block item in nanoseconds, only applicable when streamingMode=CONSTANT_RATE | `1_500_000` | -| `maxBlockItemsToStream` | exit condition for the simulator and the circular implementations such as `BlockAsDir` or `BlockAsFile` implementations | `10_000` | -| `streamingMode` | can either be `CONSTANT_RATE` or `MILLIS_PER_BLOCK` | `CONSTANT_RATE` | -| `millisecondsPerBlock` | if streamingMode is `MILLIS_PER_BLOCK` this will be the time to wait between blocks in milliseconds | `1_000` | -| `blockItemsBatchSize` | the number of block items to send in a single batch, however if a block has less block items, it will send all the items in a block | `1_000` | - -### GeneratorConfig -Uses the prefix `generator` so all properties should start with `generator.` -| Key | Description | Default Value | -|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------| -| `generationMode` | The desired generation Mode to use, it can only be `DIR` or `AD_HOC` | `DIR` | -| `folderRootPath` | If the generationMode is DIR this will be used as the source of the recording to stream to the Block-Node | `` | -| `managerImplementation` | The desired implementation of the BlockStreamManager to use, it can only be `BlockAsDirBlockStreamManager`, `BlockAsFileBlockStreamManager` or `BlockAsFileLargeDataSets` | `BlockAsFileBlockStreamManager` | -| `paddedLength` | on the `BlockAsFileLargeDataSets` implementation, the length of the padded left zeroes `000001.blk.gz` | 36 | -| `fileExtension` | on the `BlockAsFileLargeDataSets` implementation, the extension of the files to be streamed | `.blk.gz` | - - - -### GrpcConfig -Uses the prefix `grpc` so all properties should start with `grpc.` - -| Key | Description | Default Value | -|-----------------|----------------------------|---------------| -| `serverAddress` | The host of the Block-Node | `localhost` | -| `port` | The port of the Block-Node | `8080` | +Refer to the [Configuration](docs/configuration.md) for configuration options. -## Building the Project +## Quickstart -To build the project, run the following command: - -```sh -./gradlew :simulator:build -``` - -## Running the Project - -Usually you will want to run a Block-Node server before the simulator, for that you can use the following commnad: - -```sh - ./gradlew :server:run -``` -However we recommend running the block-node server as a docker container: -```sh -./gradlew :server:build :server:createDockerImage :server:startDockerContainer -``` - -Once the project is built, you can run it using the following command: - -```sh -./gradlew :simulator:run -``` +Refer to the [Quickstart](docs/quickstart.md) for a quick guide on how to get started with the application. diff --git a/simulator/docs/configuration.md b/simulator/docs/configuration.md new file mode 100644 index 000000000..dfa78fbb5 --- /dev/null +++ b/simulator/docs/configuration.md @@ -0,0 +1,39 @@ +# Configuration + +There are 3 configuration sets: +1. [BlockStreamConfig](#blockstreamconfig): contains the configuration for the Block Stream Simulator logic. +1. [GeneratorConfig](#generatorconfig): contains the configuration for the Block Stream Simulator generation module. +1. [GrpcConfig](#grpcconfig): contains the configuration for the gRPC communication with the Block-Node. + +## BlockStreamConfig + +Uses the prefix `blockStream` so all properties should start with `blockStream.` + +| Key | Description | Default Value | +|:---|:---|---:| +| `delayBetweenBlockItems` | The delay between each block item in nanoseconds, only applicable when streamingMode=CONSTANT_RATE | `1_500_000` | +| `maxBlockItemsToStream` | exit condition for the simulator and the circular implementations such as `BlockAsDir` or `BlockAsFile` implementations | `10_000` | +| `streamingMode` | can either be `CONSTANT_RATE` or `MILLIS_PER_BLOCK` | `CONSTANT_RATE` | +| `millisecondsPerBlock` | if streamingMode is `MILLIS_PER_BLOCK` this will be the time to wait between blocks in milliseconds | `1_000` | +| `blockItemsBatchSize` | the number of block items to send in a single batch, however if a block has less block items, it will send all the items in a block | `1_000` | + +## GeneratorConfig + +Uses the prefix `generator` so all properties should start with `generator.` + +| Key | Description | Default Value | +|:---|:---|---:| +| `generationMode` | The desired generation Mode to use, it can only be `DIR` or `AD_HOC` | `DIR` | +| `folderRootPath` | If the generationMode is DIR this will be used as the source of the recording to stream to the Block-Node | `` | +| `managerImplementation` | The desired implementation of the BlockStreamManager to use, it can only be `BlockAsDirBlockStreamManager`, `BlockAsFileBlockStreamManager` or `BlockAsFileLargeDataSets` | `BlockAsFileBlockStreamManager` | +| `paddedLength` | on the `BlockAsFileLargeDataSets` implementation, the length of the padded left zeroes `000001.blk.gz` | 36 | +| `fileExtension` | on the `BlockAsFileLargeDataSets` implementation, the extension of the files to be streamed | `.blk.gz` | + +## GrpcConfig + +Uses the prefix `grpc` so all properties should start with `grpc.` + +| Key | Description | Default Value | +|:---|:---|---:| +| `serverAddress` | The host of the Block-Node | `localhost` | +| `port` | The port of the Block-Node | `8080` | diff --git a/simulator/docs/quickstart.md b/simulator/docs/quickstart.md new file mode 100644 index 000000000..9022f548f --- /dev/null +++ b/simulator/docs/quickstart.md @@ -0,0 +1,60 @@ +# Quickstart of the Simulator + +## Table of Contents + +1. [Configuration](#configuration) +1. [Running locally](#running-locally) + 1. [Build the Simulator](#build-the-simulator) + 1. [Run the Server first](#run-the-server-first) + 1. [Run the Simulator](#run-the-simulator) + 1. [Run the Simulator with Debug](#run-the-simulator-with-debug) + +## Configuration + +Refer to the [Configuration](configuration.md) for configuration options. + +## Running locally: + +- Simulator subproject qualifier:`:simulator` +- Assuming your working directory is the repo root + +> **NOTE:** one may use the `-p` flag for `./gradlew` in order to avoid +> specifying the target subproject repeatedly on each task when running +> multiple tasks. When running only a single task, however, it is +> recommended to use the project qualifier (i.e. `:simulator:`) for +> both simplicity and clarity. + +### Build the Simulator + +> **NOTE:** if you have not done so already, it is +> generally recommended to build the entire repo first: +> ```bash +> ./gradlew clean build -x test +> ``` + +1. To quickly build the Simulator sources (without running tests), do the following: + ```bash + ./gradlew -p simulator clean build -x test + ``` + +### Run the Server first + +Usually, you would want to run the [Server](../../server/README.md) first, refer to the +[Quickstart of the Server](../../server/docs/quickstart.md) for a quick guide on how to +get started with the application. + +### Run the Simulator + +1. To start the Simulator, do the following: + ```bash + ./gradlew :simulator:run + ``` + +### Run the Simulator with Debug + +1. To start the Simulator with debug enabled, do the following: + ```bash + ./gradlew :simulator:run --debug-jvm + ``` + +1. Attach your remote jvm debugger to port 5005. diff --git a/tools/README.md b/tools/README.md index 5def135d9..d479002fc 100644 --- a/tools/README.md +++ b/tools/README.md @@ -1,45 +1,63 @@ # Command Line Tools for Block Nodes & Streams +## Table of Contents + +1. [Overview](#overview) +1. [Subcommands](#subcommands) + 1. [The `json` Subcommand](#the-json-subcommand) + 1. [The `info` Subcommand](#the-info-subcommand) +1. [Running from command line](#running-from-command-line) + +## Overview + This subproject provides command line tools for working with block stream files and maybe other things in the future. It uses [picocli](https://picocli.info) to provide a command line interface which makes it easy to extend and add new subcommands or options. -## Running from command line -You can run through gradle with the `tools:run` task. For example, to see the help for the `info` subcommand, you can -run: - -`./gradlew -q tools:run --args="info --help"` - ## Subcommands + The following subcommands are available: -- **json** Converts a binary block stream to JSON -- **info** Prints info for block files +- `json` - Converts a binary block stream to JSON +- `info` - Prints info for block files + +### The `json` Subcommand -# JSON Subcommand Converts a binary block stream to JSON `Usage: subcommands json [-t] [-ms=] [...]` **Options:** + - `-ms ` or `--min-size=` - - Filter to only files bigger than this minimum file size in megabytes + - Filter to only files bigger than this minimum file size in megabytes + - `-t` or `--transactions` - - expand transactions, this is no longer pure json conversion but is very useful making the + - expand transactions, this is no longer pure json conversion but is very useful making the transactions human-readable. + - `...` - - The block files or directories of block files to convert to JSON + - The block files or directories of block files to convert to JSON + +### The `info` Subcommand -# Info Subcommand Prints info for block files `Usage: subcommands info [-c] [-ms=] [-o=] [...]` **Options:** + - `-c` or `--csv` - - Enable CSV output mode (default: false) + - Enable CSV output mode (default: false) + - `-ms ` or `--min-size=` - - Filter to only files bigger than this minimum file size in megabytes + - Filter to only files bigger than this minimum file size in megabytes + - `-o ` or `--output-file=` - - Output to file rather than stdout + - Output to file rather than stdout + - `...` - - The block files or directories of block files to print info for + - The block files or directories of block files to print info for + +## Running from command line + +Refer to the [Quickstart](docs/quickstart.md) for a quick guide on how to run the tools CLI. diff --git a/tools/docs/quickstart.md b/tools/docs/quickstart.md new file mode 100644 index 000000000..b327ee67e --- /dev/null +++ b/tools/docs/quickstart.md @@ -0,0 +1,41 @@ +# Quickstart of the Tools + +## Table of Contents + +1. [Running locally](#running-locally) + 1. [Build the Tools](#build-the-tools) + 1. [Run the Tools](#run-the-tools) + +## Running locally: + +- Tools subproject qualifier: `:tools` +- Assuming your working directory is the repo root + +> **NOTE:** one may use the `-p` flag for `./gradlew` in order to avoid +> specifying the target subproject repeatedly on each task when running +> multiple tasks. When running only a single task, however, it is +> recommended to use the project qualifier (i.e. `:tools:`) for +> both simplicity and clarity. + +### Build the Tools + +> **NOTE:** if you have not done so already, it is +> generally recommended to build the entire repo first: +> ```bash +> ./gradlew clean build -x test +> ``` + +1. To quickly build the Tools sources (without running tests), do the following: + ```bash + ./gradlew -p tools clean build -x test + ``` + +### Run the Tools + +1. To run the Tools, do the following: + ```bash + # Here is an example of running the info command with the help option, simply + # replace `info --help` with the desired command and options to run the tools + # quickly using the `./gradlew run` task. + ./gradlew -q :tools:run --args="info --help" + ```