With Anthos Config Management, you can create a common configuration for all administrative policies that apply to your Kubernetes clusters both on-premises and in the cloud. With the Anthos Config Management Operator deployed in your Kubernetes cluster, it will continuously watch and deploy the appropriate changes so that your desired state is always reflected.
This pattern of deploying an operator inside each cluster that continuously monitors a central git repository and implements that desired state is a powerful way to distribute configuration changes and policies across a fleet of GKE clusters.
This demo will show you how to configure a central git repository, deploy the Operator to the base GKE cluster, and push changes that the Operator will act on automatically.
- Prerequisites
- Initializing the Cloud Source Repository
- Installing Anthos Configuration Management
- Next Steps
- Teardown
- Troubleshooting
- Relevant Material
Deploy the base cluster in the target project as per the instructions in the top-level README and configure your terminal to access the private cluster.
Download the nomos CLI tool from Anthos Configuration Management.
For MacOS, choose darwin_amd64 and for Cloud Shell choose linux_amd64. After downloading the binary, configure it to be executable:
chmod +x nomosNext, copy the nomos binary to a location in your $PATH. On Linux/Cloud Shell/MacOS, you can run:
sudo cp nomos /usr/local/bin/nomosVerify Nomos has been installed into your $PATH:
which nomos
/usr/local/bin/nomosOn MacOS, you can install kustomize with the Homebrew package manager:
brew install kustomizeCloud Shell/Linux:
opsys=linux # or darwin, or windows
# Get the latest version of kustomize
curl -s https://api.github.com/repos/kubernetes-sigs/kustomize/releases/latest |\
grep browser_download |\
grep $opsys |\
cut -d '"' -f 4 |\
xargs curl -L -o kustomizeCopy kustomize to a location in your $PATH:
sudo cp kustomize /usr/local/bin/kustomizeMake the kustomize binary executable:
sudo chmod +x /usr/local/bin/kustomizeVerify kustomize is working correctly:
which kustomize
/usr/local/bin/kustomizeIn Cloud Shell, kubectl is already installed, so you may move on to the next section. On MacOS:
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectlMake the kubectl binary executable:
chmod +x kubectlMove the binary into a directory in your $PATH:
sudo mv kubectl /usr/local/bin/kubectlTest to ensure the version you installed is up-to-date:
kubectl versionOnce you have confirmed access to your private cluster, you need to clone a git repository and initialize it using the nomos CLI.
Change to the anthos directory:
cd anthosSetup the values for your repository ID, Project and account name:
REPO="anthos-demo"
PROJECT=$(gcloud config list --format 'value(core.project)' 2>/dev/null)
ACCOUNT=$(gcloud config list --format 'value(core.account)' 2>/dev/null)Verify these settings are set and correct:
echo $REPO
echo $PROJECT
echo $ACCOUNTCreate a cloud source repo:
gcloud source repos create "${REPO}"Clone the repository and change directory into the cloned repo:
gcloud source repos clone "${REPO}"
cd "${REPO}"Initialize the repository using nomos:
nomos initAdd all new/changed files to the cloned repo, commit with a message, and push the change to the master branch:
git add .
git commit -m 'Adding initial files for nomos'
git pushThis creates the basic directory structure used by the Anthos Configuration Management operator. Specifically, this creates the ./system, ./cluster, ./clusteregistry, and ./namespaces directories.
Anthos Config Management keeps your enrolled clusters in sync using kubernetes manifests that are checked into a git source control repository. A kubernetes manifest is a YAML or JSON file that is stored in your repository and contains the same types of configuration details that you can manually apply to a cluster using the kubectl apply command. This topic covers how configs work, how to write them, and how Anthos Config Management applies them to your enrolled clusters.
When you create a configuration manifest, you need to decide the best location in the repository and the fields to include. The location of a configuration manifest in the repository determines which cluster(s) it applies to.
- Configuration manifests for cluster-scoped objects except for
namespacesare stored in the./clustersdirectory of the repo. - Configuration manifests for
namespacesandnamespace-scoped objects are stored in the./namespacesdirectory of the repo. - Configuration manifests for Anthos Config Management components are stored in the
./systemdirectory of the repo. - The configuration manifest for the Config Management Operator is not stored directly in the repository and is not synced.
One of the simplest examples is to create a namespace. In this example, it's named audit. From inside the current repository, copy the files from the example folder into your current folder:
cp -R ../anthos-config-example/namespaces/audit ./namespaces/Add the SSH key materal to the .gitignore to prevent getting checked into source control:
cat <<EOT >> .gitignore
anthos-demo-key
anthos-demo-key.pub
config-management-operator.yaml
config-management.yaml
EOTAdd all new/changed files to the cloned repo, commit with a message, and push the change to the master branch:
git add .
git commit -m 'Adding namespace'
git pushTo enroll a cluster in Anthos Config Management, you deploy the Anthos "Operator" manifest, create the git-creds secret, and finally configure the Operator. Note that currently, Pod Security Policies are not supported in conjunction with Anthos Config Management.
After ensuring that you meet all the prerequisites, you can deploy the Operator by downloading and applying a YAML manifest.
Download the latest version of the Operator CRD using the following command:
gsutil cp gs://config-management-release/released/latest/config-management-operator.yaml config-management-operator.yamlYou may already have kubectl with the proxy settings aliased, but you can ensure this is set in the current terminal session:
alias k="HTTPS_PROXY=localhost:8888 kubectl"Apply the manifest which installs the necessary components and starts the Operator:
k apply -f config-management-operator.yaml
customresourcedefinition.apiextensions.k8s.io/configmanagements.addons.sigs.k8s.io created
clusterrolebinding.rbac.authorization.k8s.io/config-management-operator created
clusterrole.rbac.authorization.k8s.io/config-management-operator created
serviceaccount/config-management-operator created
deployment.apps/config-management-operator created
namespace/config-management-system createdThe Operator needs read-only access to your Git repository (the repo) so it can read the configurations committed to the repository and automatically apply them to your clusters. If credentials are required, they are stored in the git-creds Kubernetes secret on each enrolled cluster.
You will want to create an SSH keypair to allow the Operator to authenticate to your Git repository. This is necessary if you need to authenticate to the repository in order to clone it or read from it.
The following command creates 4096-bit RSA key. Replace [GIT REPOSITORY USERNAME] and /path/to/[KEYPAIR-FILENAME] with the values you want the Operator to use to authenticate to the repository.
ssh-keygen -t rsa -b 4096 \
-C "${ACCOUNT}" \
-N '' \
-f ./anthos-demo-keyAdd the following SSH public key to the Cloud Source Repository. You can obtain the contents of the key-file by using a cat command:
cat anthos-demo-key.pubAdd the contents of the public SSH Key to your Cloud Source Repository SSH Keys configuration via the UI/Console https://source.cloud.google.com/user/ssh_keys. Click Register SSH Key, use anthos-demo for the key name, copy/paste in the contents of the public key, and click Register.
Add the private key to a new secret in the cluster:
k create secret generic git-creds \
--namespace=config-management-system \
--from-file=ssh=./anthos-demo-key
secret/git-creds createdFinally, delete the private key from the local disk (e.g. rm anthos-demo-key) or otherwise take appropriate measures to protect it.
To configure the behavior of the Operator, you create a configuration file for the ConfigManagement CustomResource, then apply it using kubectl.
The spec field contains configuration details such as the location of the repository and secretType to use. Avoid making changes to configuration values outside the spec field. The following command creates a working Operator configuration manifest:
cat > config-management.yaml <<EOF
apiVersion: addons.sigs.k8s.io/v1alpha1
kind: ConfigManagement
metadata:
name: config-management
namespace: config-management-system
spec:
# clusterName is required and must be unique among all managed clusters
clusterName: demo-cluster
git:
syncRepo: ssh://${ACCOUNT}@source.developers.google.com:2022/p/${PROJECT}/r/${REPO}
syncBranch: master
secretType: ssh
policyDir: "."
EOFApply the configuration manifest to the cluster:
k apply -f config-management.yaml
configmanagement.addons.sigs.k8s.io/config-management createdYou can use the nomos status command to check if the Operator is installed successfully and is reporting the correct status. Please note that like kubectl, the nomos CLI also needs to communicate to the private cluster over the proxy. A similar use of a shell alias can reduce the amount of typing:
alias n="HTTPS_PROXY=localhost:8888 nomos"
n statusA valid installation with no problems has a status of PENDING or SYNCED. An invalid or incomplete installation has a status of NOT INSTALLED or NOT CONFIGURED. The output also includes any reported errors.
When the Operator is deployed successfully, it runs in a pod whose name begins with config-management-operator, in the kube-system namespace. The pod may take a few moments to initialize. Verify that the pod is running:
k -n kube-system get pods | grep config-managementIf the pod is running, the command's response is similar (but not identical) to the following:
config-management-operator-6f988f5fdd-4r7tr 1/1 Running 0 26sYou can also verify that the config-management-system namespace exists:
k get namespaces | grep 'config-management-system'
config-management-system Active 1mReturn to the top-level README to begin working on another topic area.
The steps completed above to install and configure a Cloud Source Repository and install the Anthos Configuration Management Operator inside the cluster are necessary prerequisites for several of the other topic areas in this repository. If you would like to continue working on other topics, refer to the next steps.
If you are completely finished working with the contents of this repository and want to remove the resources created above, delete the Cloud Source Repository:
gcloud source repos delete ${REPO}
Delete "anthos-demo" in project "my-project-id" (Y/n)? y
Deleted [anthos-demo].Remove the SSH Public Key by visiting https://source.cloud.google.com/user/ssh_keys and deleting the anthos-demo SSH Key.
Follow the teardown steps in the top-level README to remove the cluster and supporting resources.
To remove the shell alias nomos, run:
unalias nIt is possible to have an invalid configuration that isn't detected right away, such as a missing or invalid git-creds secret. For troubleshooting steps, refer to the Valid but incorrect ConfigManagement object section in the official troubleshooting documentation.
Note, this is not an officially supported Google product