ec2 architecture and guide

docs/self-managed/reference-architecture/manual/manual.md

@@ -0,0 +1,123 @@
+---
+id: manual
+title: "Manual JAR deployment overview"
+sidebar_label: Manual JAR
+description: "Camunda 8 Manual (Java) deployment Reference architecture home "
+---
+
+<!-- Moving target, may be renamed, etc. -->
+
+This reference architecture provides guidance on deploying Camunda 8 Self-Managed as a standalone Java application. This deployment method is ideal for users who prefer manual deployment on bare metal servers or virtual machines (VMs), offering full control over the environment and configuration. It is particularly suited for scenarios with specific infrastructure requirements or highly customized setups.
+
+:::note
+This method of deployment requires a solid understanding of infrastructure, networking, and application management. Consider evaluating your [deployment platform options](../reference-architecture.md) based on your familiarity and need. If you prefer a simpler and managed solution, [Camunda 8 SaaS](https://camunda.com/platform/) can significantly reduce maintenance efforts, allowing you to focus on your core business needs.
+:::
+
+## Key features
+
+- **Single application JAR**: Starting from Camunda 8.7, all core components (Zeebe, Tasklist, Operate, Optimize, and Identity) are bundled into a single JAR file. This simplifies deployment by reducing the number of artifacts to manage.
+- **Full control**: Users are responsible for all aspects of deployment, including installation, configuration, scaling, and maintenance. This offers maximum flexibility for custom environments.
+
+Other deployment options, such as containerized deployments or managed services, might offer more convenience and automation. However, VM based deployment gives you the flexibility to tailor the deployment to your exact needs, which can be beneficial for regulated or highly customized environments.
+
+For documentation on the orchestration cluster, Web Modeler and Console separation, refer to the [reference architecture overview](/self-managed/reference-architecture/reference-architecture.md#orchestration-cluster-vs-web-modeler-and-console).
+
+## Reference implementations
+
+This section includes deployment reference architectures for manual setups:
+
+- [Amazon EC2 deployment](/self-managed/setup/deploy/amazon/aws-ec2.md) - a standard production setup with support for high availability.
+
+## Considerations
+
+- This overview page focuses on deploying the [orchestration cluster](/self-managed/reference-architecture/reference-architecture.md#orchestration-cluster), the single JAR compromised of Identity, Operate, Optimize, Tasklist, and Zeebe, as well as the Connectors runtime. Web Modeler and Console deployments are not included.
+- General guidance and examples focuses on **unix** users, but can be adapted by Windows users with options like [WSL](https://learn.microsoft.com/en-us/windows/wsl/install) or included `batch` files.
+- The Optimize importer is not highly available and must only run once within the whole setup.
+
+## Architecture
+
+![Single JAR](./img/manual-single.jpg)
+
+This above diagram illustrates a single-machine deployment using the single JAR package. While simple and effective for lightweight setups, scaling to multiple machines requires careful planning.
+
+Compared to the generalized architecture depicted in the [reference architecture](/self-managed/reference-architecture/reference-architecture.md#architecture), the `Optimize importer` can be enabled as part of the single JAR.
+
+### High Availability (HA)
+
+:::caution Non-HA Optimize importer
+When scaling from a single machine to multiple machines, ensure that the `Optimize importer` is enabled on only one machine and disabled on the others. Enabling it on multiple machines will cause data inconsistencies. This limitation is known and will be addressed in future updates.
+:::
+
+![HA JAR](./img/manual-ha.jpg)
+
+For high availability, a minimum of three machines is recommended to ensure fault tolerance and enable master election in case of failures. Refer to the [clustering documentation](/components/zeebe/technical-concepts/clustering.md) to learn more about the raft protocol and clustering concepts.
+
+### Components
+
+The orchestration core is packaged as a single JAR file and includes the following components:
+
+- **Zeebe**
+- **Operate**
+- **Tasklist**
+- **Optimize**
+- **Identity**
+
+The core facilitates:
+
+1. **gRPC communication**: For client workers.
+2. **HTTP endpoints**: Used by the REST API and Web UI.
+
+Both types of endpoints can be routed through a load balancer to maintain availability, ensuring that the system remains accessible even if a machine becomes unavailable. While using a load balancer is optional, it is recommended for enhanced availability and security. Alternatively, you can expose static machines, ports, and IPs directly. However, direct exposure is generally discouraged due to security concerns.
+
+Connectors expose additional HTTP(s) endpoints for handling incoming webhooks, which can also be routed through the same http load balancer.
+
+The orchestration components rely on **Elasticsearch** or **OpenSearch** as their data store.
+
+Components within the orchestration core communicate seamlessly, particularly:
+
+- **Zeebe brokers** exchange data over gRPC endpoints for efficient inter-broker communication.
+
+## Requirements
+
+Before implementing a reference architecture, review the requirements and guidance outlined below. We are differentiating between `Infrastructure` and `Application` requirements.
+
+### Infrastructure
+
+Any of the following are just suggestions for the minimum viable setup, the sizing heavily depends on your use cases and usage. It is recommended to understand the documentation on [sizing your environment](/components/best-practices/architecture/sizing-your-environment.md) and run benchmarking to confirm your required needs.
+
+#### Minimum Requirements Per Host
+
+- Modern CPU: 4 cores
+- Memory: 8 GB RAM
+- Storage: 32 GB SSD (**1,000** IOPS recommended; avoid burstable disk types)
+
+Suggested instance types from cloud providers:
+
+- AWS: [m7i](https://aws.amazon.com/ec2/instance-types/m7i/) series
+- GCP: [n1](https://cloud.google.com/compute/docs/general-purpose-machines#n1_machines) series
+
+#### Networking
+
+- Stable and high-speed network connection
+- Configured firewall rules to allow necessary traffic:
+  - **8080**: Web UI / REST endpoint.
+  - **9090**: Connector port.
+  - **9600**: Metrics endpoint.
+  - **26500**: gRPC endpoint.
+  - **26501**: Gateway-to-broker communication.
+  - **26502**: Inter-broker communication.
+- Load balancer for distributing traffic (if required)
+
+:::info Customizing ports
+Some ports can be overwritten and are not definitive, you may conduct the documentation of each component to see how it can be done, in case you want to use a different port. Or in our example `Connectors` and `Web UIs` overlap on 8080 due to which we moved connectors to a different port.
+:::
+
+### Application
+
+- Java Virtual Machine, see [supported environments](/reference/supported-environments.md) for version details.
+
+### Database
+
+- Elasticsearch / OpenSearch, see [supported environments](/reference/supported-environments.md) for version details.
+
+Our recommendation is to use an external managed offer as we will not go into detail on how to manage and maintain your database.

docs/self-managed/reference-architecture/reference-architecture.md

@@ -0,0 +1,144 @@
+---
+id: reference-architecture
+title: "Camunda 8 reference architectures"
+sidebar_label: "Overview"
+description: "Learn about the Self-Managed reference architectures and how they can help you get started."
+---
+
+Reference architectures provide a comprehensive blueprint for designing and implementing scalable, robust, and adaptable systems. The reference architectures published here offer guidance to help enterprise architects, developers, and IT managers streamline deployments and improve system reliability.
+
+## Overview
+
+Reference architectures are not a one-size-fits-all solution, and each organization has unique requirements and constraints that may necessitate modifications to the provided blueprints.
+
+While these reference architectures offer a solid foundation and best practices, they should be adapted to fit the specific needs of your project. Use them as a starting point to start your Camunda 8 implementation process, but be prepared to make adjustments to ensure they align with your goals and infrastructure.
+
+### Target users
+
+- **Enterprise architects**: To design and plan the overall system structure.
+- **Developers**: To understand the components and their interactions.
+- **IT managers**: To ensure the system meets business requirements and is maintainable.
+
+### Key benefits
+
+- **Accelerated deployment**: Predefined best practices and guidelines simplify the deployment process, reducing the time and effort required to set up a reliable workflow automation solution.
+- **Consistency**: Ensures consistency across deployments by standardizing system components and their configurations, which helps reduce the risk of errors and simplifies maintenance.
+- **Enhanced security**: Reference architectures incorporate best practices for securing Camunda 8 deployments, ensuring that sensitive data and processes are protected through standard security measures like encryption, authentication, and access controls.
+
+### Support considerations
+
+Deviations from the reference architecture are unavoidable. However, such changes will introduce additional complexity, making troubleshooting more difficult. When modifications are required, ensure they are well-documented to facilitate future maintenance and troubleshooting more quickly. Camunda publishes [supported environment](/reference/supported-environments.md) information to help you navigate supported configurations.
+
+## Architecture
+
+### Orchestration cluster vs Web Modeler and Console
+
+When designing a reference architecture, it's essential to understand the differences between an orchestration cluster and Web Modeler and Console Self-Managed. These components play crucial roles in the deployment and operation of processes, but they serve different purposes and include distinct components.
+
+#### Orchestration Cluster
+
+![Orchestration Cluster](./img/orchestration-cluster.jpg)
+
+The orchestration cluster is the core of Camunda.
+
+The included components are:
+
+- [Zeebe](/components/zeebe/zeebe-overview.md): A workflow engine for orchestrating microservices and managing stateful, long-running business processes.
+- [Operate](/components/operate/operate-introduction.md): A monitoring tool for visualizing and troubleshooting workflows running in Zeebe.
+- [Tasklist](/components/tasklist/introduction-to-tasklist.md): A user interface for managing and completing human tasks within workflows.
+- [Optimize]($optimize$/components/what-is-optimize/): An analytics tool for generating reports and insights based on workflow data.
+- [Identity](/self-managed/identity/what-is-identity.md): A service for managing user authentication and authorization.
+- [Connectors](/components/connectors/introduction.md): Pre-built integrations for connecting Zeebe with external systems and services.
+
+Each component within the orchestration cluster is part of an integrated system that works together to provide end-to-end process orchestration. These components form a unified cluster that is tightly integrated to ensure seamless communication and data flow.
+
+This design ensures that all components are in sync, working collectively to maintain consistent state management, data integrity, and smooth process orchestration across the entire cluster. This architecture ensures reliable process execution with clear boundaries between each workflow engine's operation.
+
+#### Web Modeler and Console
+
+![Web Modeler and Console](./img/management-cluster.jpg)
+
+Web Modeler and Console are designed to interact with multiple orchestration clusters. Console offers tools and interfaces for administrators to monitor clusters, and Web Modeler allows developers to create and deploy BPMN models.
+
+- [Console](/components/console/introduction-to-console.md): A central management interface for monitoring and managing multiple orchestration clusters.
+- [Web Modeler](/self-managed/modeler/web-modeler/installation.md): A web-based tool for designing and deploying workflow models to any available orchestration cluster.
+
+Additionally, Web Modeler and Console require the following:
+
+- [Identity](/self-managed/identity/what-is-identity.md): A service for managing user authentication and authorization.
+
+Unlike the orchestration cluster, Web Modeler and Console run a separate and dedicated Identity deployment. For production environments, using an external [identity provider](/self-managed/setup/guides/connect-to-an-oidc-provider.md) is recommended.
+
+### Databases
+
+Databases can be deployed as part of the Camunda clusters, but using external databases or managed services offers several advantages:
+
+- **Flexibility**: Allows you to choose the database technology that best fits your needs and existing infrastructure while choosing one of the [supported environments](/reference/supported-environments.md#component-requirements).
+- **Scalability**: External databases can be scaled independently of the Camunda components, providing better performance and resource management.
+- **Maintenance**: Simplifies the maintenance and upgrade processes, as database management can be handled separately.
+- **Compliance**: Ensures that you can adhere to specific data governance and compliance requirements.
+
+While some guides go into detail on how to deploy databases together with Camunda, the recommendation is to maintain this outside of Camunda.
+
+By decoupling databases from Camunda, you gain greater control and customization over your data storage and management strategies.
+
+### High availability (HA)
+
+High availability (HA) ensures that a system remains operational and accessible even in the event of component failures. While all components are equipped to be run in a highly available manner, some components need extra considerations when run in HA mode.
+
+<!-- TODO Describe Optimize and Connectors limitations or point to resource for more -->
+
+While high availability is one part of the increased fault tolerance and resilience, you should also consider regional or zonal placement of your workloads.
+
+If you run infrastructure on cloud providers, you are often met with different regions and zones. For ideal high availability you should consider a minimum setup of 3 zones within a region as this will guarantee that in case of a zonal failure that the remaining two workloads can still process data. For more information on how Zeebe handles fault tolerance, have a look at the [raft consensus chapter](/components/zeebe/technical-concepts/clustering.md#raft-consensus-and-replication-protocol).
+
+If running a single instance is preferred, make sure to implement [regular backups](/self-managed/operational-guides/backup-restore/backup-and-restore.md) since resilience will be limited.
+
+## Available reference architectures
+
+:::note Documentation Update in Progress
+This is a work in progress as the existing documentation is updated to provide better general guidance on the topic. The Kubernetes and Docker documentation may point to older guides.
+:::
+
+Choosing the right reference architecture depends on various factors such as the organization's goals, existing infrastructure, and specific requirements. The following guides are available to help choose and guide deployments:
+
+### Kubernetes
+
+Kubernetes is a powerful orchestration platform for containerized applications. Using a reference architecture for Kubernetes can help organizations deploy and manage their applications more effectively. It provides guidelines for setting up clusters, managing workloads, and ensuring high availability and scalability. This approach is ideal for organizations looking to leverage the benefits of containerization and self-healing capabilities.
+
+- Ideal for organizations adopting containerization and microservices (see [Cloud Native computing foundation](https://www.cncf.io/)).
+- Suitable for dynamic scaling and high availability.
+- Best for teams with experience in managing containerized environments.
+- A steeper learning curve but offers scalable and highly resilient platform.
+
+For more information and guides, see the reference for [Kubernetes](/self-managed/setup/install.md).
+
+<!-- TODO add link or card for AWS ref arch -->
+
+### Containers
+
+Containers, such as Docker, offer a middle ground between the manual JAR and Kubernetes approaches. They provide a lightweight, portable, and consistent runtime environment, making it easier to develop, test, and deploy applications across different environments. Containers encapsulate an application and its dependencies, ensuring that it runs reliably regardless of where it is deployed.
+
+- Advisable as a middle ground between manual JAR and Kubernetes. Profit from containerization while not having the whole overhead of Kubernetes.
+- Containers can run on any system that supports the container runtime, ensuring consistency across development, testing, and production environments.
+- Each container runs in its own isolated environment, which helps prevent conflicts between applications and improves security.
+- Containers can be easily scaled up or down to handle varying workloads, providing flexibility in resource management.
+
+For more information and guides, see the reference for [containers](/self-managed/setup/deploy/other/docker.md).
+
+### Manual JAR (bare metal/virtual machines)
+
+For organizations that prefer traditional infrastructure, reference architectures for bare metal or virtual machines (VMs) offer a structured approach to system deployment. These architectures provide best practices for setting up physical servers or VMs, configuring networks, and managing storage using Infrastructure as Service cloud providers. They are suitable for environments where containerization or use of Kubernetes services may not be feasible.
+
+- Suitable for organizations requiring use of IaaS, bare metal, and other traditional infrastructures.
+- Ideal for traditional setups needing highly customized security, strict data residency, or industry-specific regulatory compliance.
+- Applicable for high availability but requires more detailed planning.
+- Best for teams with expertise in managing physical servers or virtual machines.
+
+For more information and guides, see the reference for [manual setups](./manual/manual.md).
+
+<!-- TODO add link or card for AWS EC2 -->
+
+### Local development
+
+While the above options are suitable for trying out Camunda 8 locally, [Camunda 8 Run](/self-managed/setup/deploy/local/c8run.md) provides a simplified, developer-focused experience.