Skip to content

Commit

Permalink
enhancing the reliability section
Browse files Browse the repository at this point in the history
  • Loading branch information
hamza-m-masood committed Dec 23, 2024
1 parent 3ca1daa commit daea8bb
Showing 1 changed file with 88 additions and 16 deletions.
104 changes: 88 additions & 16 deletions docs/self-managed/operational-guides/production-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,7 @@ If you would like more information on configuring ILM policy. Please refer to [o

## Scaling and Performance

At this point you should already have a solid base to run your platform in a production setting. The rest of this guide gives you general Kubernetes based guidance on configuration for long-term maintenance.
At this point you should already have a solid base to run your platform in a production setting. The rest of this guide gives you general Kubernetes based guidance on configuration of the Camunda Helm Chart for long-term maintenance.

Here are some points to keep in mind when considering scalability:

Expand Down Expand Up @@ -301,27 +301,99 @@ core:
maxUnavailable: 1
```

- Version Management: Stay on a stable Camunda and Kubernetes version and follow Camunda’s release notes for security patches or critical updates.
- Secrets should be created prior to installing the helm chart so they can be referenced as existingSecrets when installing the helm chart.
- Familiar with upgrades. Ideally, customers should have already performed an upgrade in the lower environment before going to production.
- Containers do not store any state in their local filesystem. State should be stored in PVs
- Mount Secrets as volumes, not enviroment variables
- Always asses the kubernetes object version and be weary of alpha or beta versions.
- Namespaces have ResourceQuotas
- Version Management: Stay on a stable Camunda and Kubernetes version. Follow Camunda’s release notes for security patches or critical updates. A list of our supported versions Camunda Helm Charts can be found on the [version matrix](https://helm.camunda.io/camunda-platform/version-matrix/)
- Secrets should be created prior to installing the helm chart so they can be referenced as existing secrets when installing the helm chart.
Please refer to the [Kuberentes documentation](https://kubernetes.io/docs/concepts/configuration/secret/) on how to create a secret.

If you would not like to create a number of secrets before installing the Camunda Helm Chart then you can optionally add the following to your `production-values.yaml`:

```yaml
global:
secrets:
autoGenerated: true
name: "camunda-credentials"
```

A secret called `camunda-credentials` will be generated. It will include all the needed secret values for the Camunda Helm Chart.

:::note
The `camunda-credentials` generated secret will not be deleted if the helm chart is uninstalled
:::

- When upgrading the Camunda Helm Chart, make sure to read the [upgrade guide](http://localhost:3000/docs/next/self-managed/operational-guides/update-guide/introduction/) before upgrading and perform the upgrade on a test environment first before attempting in production.
- Make sure to not store any state or important, long term business data in the local file system of the container. A pod is transient, if the pod is restarted then the data will get whiped. It is better to create a volume and volume mount instead. Here is some eaxmple configuration for the core component to create persistent storage:

```yaml
core:
extraVolumes:
extraVolumes:
- name: persistent-state
emptyDir: {}
extraVolumeMounts:
- name: persistent-state
mountPath: /mount
```

<!-- This seems very specific to the application. I might remove this: -->
<!-- - Mount Secrets as volumes, not enviroment variables -->

- It is recommended to set a memory and resource quota for your namespace. Please refer to the [Kubernetes documenation](https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/quota-memory-cpu-namespace/) to do so.

## Security Guidelines

Here are some points to keep in mind when considering security:

- Disable auto-mounting of the default ServiceAccount
- ServiceAccount tokens are for applications and controllers only. End users should not be given these tokens
- Enable network policies
- Maybe link this to customer: https://github.com/ahmetb/kubernetes-network-policy-recipes
- Enable Pod Security Policies
- Use read-only root filesystem
- To limit unnecessary permissions, and reduce risk of token exploitations, it is recommended to disable auto-mounting of the default Service Account. It is possible to do so by adding the following to your `production-values.yaml`:

```yaml
identity:
serviceAccount:
enabled: false
console:
serviceAccount:
enabled: false
webModeler:
serviceAccount:
enabled: false
connectors:
serviceAccount:
enabled: false
core:
serviceAccount:
enabled: false
optimize:
serviceAccount:
enabled: false
```

You should only enable the auto-mounting of a service account token when the application explicitly needs access to the Kubernetes API server or you have created a service account with the exact permissions required for the application and bound it to the pod.

- If you have a usecase for enabling [Network Policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) then it is recommended to do so.
<!--Maybe link this to customer: https://github.com/ahmetb/kubernetes-network-policy-recipes-->
- It is possible to have a pod security standard that is suitable to the security constraints you might have. This is possible through modifying the Pod Security Admission. Please refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/security/pod-security-admission/) in order to do so.
- By default, the Camunda Helm Chart, uses a read-only root file-system configuration for the pod. It is recommended to keep this default. No addition needs to be made in your `production-values.yaml`.
- Disable privelaged containers
- Only allow deploying containers only from known registries: https://blog.openpolicyagent.org/securing-the-kubernetes-api-with-open-policy-agent-ce93af0552c3#3c6e
- Use approved domain names for the ingress hostname: https://www.openpolicyagent.org/docs/latest/kubernetes-tutorial/#4-define-a-policy-and-load-it-into-opa-via-kubernetes
- It is possible to modify either the `containerSecurityContext` or the `podSecurityContext`. For example, here is a configuration for the core component that can be added to your `production-values.yaml`:

```yaml
podSecurityContext:
runAsNonRoot: true
fsGroup: 1001
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1001
seccompProfile:
type: RuntimeDefault
```

- It is best to only pull from a private registry such as [Amazon ECR](https://aws.amazon.com/ecr/) and not pull directly from Docker Hub. This is because it will increase the control you have over the images, will save you from rate limits, and increase performance and reliability. Furthermore, it is possible to restrict your cluster to only pull from known registries, this can be done using [Open Policy Agent](https://blog.openpolicyagent.org/securing-the-kubernetes-api-with-open-policy-agent-ce93af0552c3#3c6e), for example.
- Open Policy Agent can also be used to have a [whitelist for ingress hostnames](https://www.openpolicyagent.org/docs/latest/kubernetes-tutorial/#4-define-a-policy-and-load-it-into-opa-via-kubernetes).

## Observability and Monitoring

Expand Down

0 comments on commit daea8bb

Please sign in to comment.