Oldest Kubernetes version supported: 1.15
⚠️ Kubernetes version 1.20.x is not supported! This Kubernetes issue prevents the controller from running properly on Kubernetes 1.20.x. The fix was released with Kubernetes 1.21. Kubernetes versions earlier than 1.20 are still supported.
Datadog runs regular chaos experiments to test the resilience of our distributed cloud applications hosted in Kubernetes. The Chaos Controller facilitates automation of these experiments by simulating common "disruptions" including but not limited to: poor network quality, exhaustion of computational resources, or unexpected node failures. All you need to do to get started is define a yaml file which contains all of the specifications needed by our custom Kubernetes Resource to run the preferred disruption Kind.
At Datadog, we use the Chaos Controller to run Gamedays where we intentionally introduce failures into staging and production systems to test the resilience of our critical applications.
The Chaos Controller allows us to simulate a variety of small and large scenarios (and combination of scenarios) such as a network degradation across an AZ which limits access to critical DNS resolvers (top) or a bad configuration triggering a “thundering herd” on Consul's key-value store (bottom).
Chaos Engineering is necessarily different from system to system. We encourage you to try out this tool, and extend it for your own use cases. If you want to run the source code locally to make and test implementation changes, visit the Contributing Docs which explain how to spin up chaos-controller source code on Minikube. By the way, we welcome Pull Requests.
If you do not plan to alter source code, you can install Chaos Controller to your existing Kubernetes environments by running:
kubectl apply -f https://raw.githubusercontent.com/DataDog/chaos-controller/main/chart/install.yaml
This install.yaml is generated for each new release and will always point to the latest stable version of the controller in Docker Hub.
If you use Helm you can install all the resources as follows:
helm install -f values.yaml chaos-controller ./chart
📣 Datadog engineers: please reach out to Chaos Engineering before deploying to a new cluster! 🙏 🙇
⚠️ cert-manager must be installed for the admission controller to get a self-signed certificate.
📖 The Advanced Installation Docs contain flags to customize webhooks, annotate injector pods, etc.
Disruptions are built as short-living resources which should be manually created and removed once your experiments are done. They should not be part of any application deployment. The Disruption resource is immutable. Once applied, you can't edit it. If you need to change the disruption definition, you need to delete the existing resource and to re-create it.
Getting started is as simple as creating a Kubernetes resource:
apiVersion: chaos.datadoghq.com/v1beta1
kind: Disruption
metadata:
name: node-failure
namespace: chaos-demo
spec:
selector: # a label selector used to target resources
app: demo-curl
count: 1 # the number of resources to target
nodeFailure:
shutdown: false # trigger a kernel panic on the target nodeTo disrupt your cluster, run kubectl apply -f <disruption_file.yaml>. You can clean up the disruption with kubectl delete -f <disruption_file>.yaml. For your safety, we recommend you get started with the dry-run mode enabled.
Check out Chaosli if you want some help understanding/creating disruption configurations.
📖 The features guide contains usecases and sample disruption files!
📖 The development guide contains tips for developing locally!
The Chaos Controller allows you to disrupt your Kubernetes infrastructure through various means including but not limited to: bringing down resources you have provisioned and preventing critical data from being transmitted between resources. The use of Chaos Controller on your production system is done at your own discretion and risk.