Configurator Architecture v0.0.2
ConfigMaps and Secrets in Kubernetes play an important role in the uptime and consistent behaviour of an application. When a user updates the ConfigMap contents, unlike deployments, Kubernetes does not create a new version of the ConfigMap and does not induce a rolling update on the deployments using those ConfigMaps. Due to this, an application using a ConfigMap does not know that the contents of the ConfigMaps have changed. This can lead to inconsistent and unpredictable behaviour of the application. Some of the present day strategies make use of ConfigMap content hash as an annotation in the deployment. Thus, when the ConfigMap content changes, the hash value changes and the annotation in the deployment changes. This automatically induces a rolling update on the deployment. When the applications get redeployed, they become aware of the ConfigMap changes. However, if a rollback of the deployment is desired, there is no way to rollback to the previous contents of the ConfigMap.
Configurator is a version control and a sync service that keeps Kubernetes ConfigMaps and Secrets in sync with the deployments.
Configurator achieves consistency and predictability by versioning the ConfigMaps and keeping these versions in sync with the deployment versions. Thus a state of deployment is always a combination of the ConfigMap version and a deployment version. If a roll back is performed, then the deployment is rolled back to a deployment version referencing a ConfigMap version.
In Configurator v0.0.1, the user was expected to maintain the configurations in a custom resource called CustomConfigMap (CCM) and not update the ConfigMap directly. This workflow was not GitOps friendly and expected the users to follow new conventions.
With v0.0.2, Configurator has been revamped to work seamlessly with GitOps workflows. In v0.0.2, Configurator heavily relies on annotations to keep the ConfigMap contents in sync with the deployment revisions and works well for both rolling updates and rollbacks. Users can directly edit the contents of the ConfigMaps. When a ConfigMap content changes, Configurator automatically detects those changes and creates a custom resource of type CustomConfigMap (CCM) with a postfix. CCM with a postfix acts like a ConfigMap revision. Configurator then copies the modified contents of the ConfigMap in to the CCM resource. In parallel, Configurator adds/modifies annotations across ConfigMaps, CCMs and deployments to maintain the dependency across these resources. A change in these annotation triggers a rolling update on deployments or keeps the contents of the ConfigMap in sync with the deployment version. Let us dive in and explore the different scenarios and how Configurator works.
Through out this document, we will be using abbreviations for simplicity.
| Abbreviation | Full form |
|---|---|
| CM | ConfigMap |
| CCM | CustomConfigMap |
Though the document describes the use cases in the context of ConfigMaps, the use cases apply to Secrets as well. To version control the Secrets, Configurator creates a CustomSecret similar to CustomConfigMap.
| Resource | Schema | Value | Purpose |
|---|---|---|---|
| ConfigMap | metadata.annotations |
customConfigMap-name: <ccm-name>
customConfigMap-version: <ccm-version>
deployments: deploy1, deploy2
updateMethod: ingnoreWhenShared |
customConfigMap-name specifies the corresponding CCM. customConfigMap-version specifies the current version of the CCM. Configurator keeps the contents of the CM in sync with the current CCM version. deployments specifies the list of deployments making use of this ConfigMap. updateMethod specifies whether the ConfigMap contents need to be updated during a deployment rollback and when the ConfigMap is used across multiple deployments. By default, it assumes ignoreWhenShared which does not update the contents of a shared ConfigMap, when a deployment using it is rolled back. |
| CustomConfigMap | metadata.annotations |
customConfigMapVersion: z70v7 |
customConfigMapVersion specifies the hash value of the CCM content, which in turn is used as a version. |
| CustomConfigMap | metadata.labels |
current: "true"
latest: "true"
name: demo-config |
latest and current specifies whether the CCM is the latest version and the current version. CCM is set to latest, if it is created as a result of the last CM edit by the user. Current specifies if CM holds the contents from this CCM. Sometimes current version may not be the latest. If the user initiates a rollback on a deployment, then an older version of CCM might be copied to CM and the older version becomes the current. |
| CustomConfigMap | metadata.ownerReferences |
References the CM to which this CCM belongs to. This is used by Kubernetes to automatically purge dependent CCMs when a CM is deleted. | |
| Deployment | spec.template.metadata.annotations |
<cm-name>: <ccm-version> | a list of all CMs used in the deployment and their corresponding versions. Helps to filter the deployments based on the CM names |
Configurator makes use of different controllers to perform background tasks and to react to user initiated events.
-
Configurator Controller : Gets triggered during create and update of CM. Creates CCM with CM contents, annotates deployments with CCM version. When a CM is updated, it updates the deployment dependencies - ie., if the CM is used in a deployment, it annotates the CM with deployment dependencies.
-
Configurator Webhook Controller : exposes API Endpoints for Admission Controllers. For more information on admission controllers, please check this
.
- /deploycontroller and /stscontroller endpoints, kind - MutatingWebhookConfiguration. Annotates deployments with CCM name and version during deployment create and update. When a deployment is create or updated, it updates the deployment dependencies - ie., if a CM is used in the deployment, it annotates the CM with deployment dependencies.
- /podcontroller (pod) validation admission controller, kind - ValidatingWebhookConfiguration. Keeps CM contents in sync with the CCM version in the pod.
Mutation admission controller (for deployments & statefulSets) - triggered when a deployment or a statefulset is created/updated. Validation admission controller - pod creation - triggered when a pod is created during a rolling update or a rollback.
Admission controllers call Configurator Webhook Controller APIs endpoints.
As soon as the Configurator controller is deployed in a cluster, the init process kicks in. It looks for all the ConfigMaps across all the namespaces and then creates CCMs with a postfix.
When a user creates a new CM, Configurator creates a corresponding CCM and adds the annotations across CM and CCM.
When a user updates a CM, it creates a new version of the CCM, annotates the CM, CCM and the deployments making use of the CM. Updating the deployment with a CM annotation triggers a rolling update and creates new pods.
Performing a rolling update on the deployment, creates new pods as well. To differentiate whether the pods were created due to CM update or due to deployment update, the CM contents are checked with the current version of the CCM.
When a user updates a deployment with a ConfigMap, it annotates the CM with the deployment dependency.
When a user deletes a CM, Kubernetes automatically deletes all the CCMs referencing the CM based on the metadata.ownerReferences in the CCMs.
Configurator controller creates a cron Job in Kubernetes. When the controller is deployed, a new Kubernetes job is created using the docker image defined in PURGE_JOB_IMAGE environment variable. This job purges any unused CCM version every 15 minutes. Unused CCM versions are decided based on their references across deployments and statefulsets and their revision history.