Chapter 3. Installation and Configuration

To configure and manage the cluster we'll use kubectl. This command use ~/.kube/config as configuration file with all the Kubernetes endpoints that you might use.

A context is a combination of a cluster and user credentials. To switch between contexts or cluster use the command:

kubectl config use-context NAME

Using Google Kubernetes Engine (GKE) requires to install Could SDK, the instructions are here. Then follow the Quickstart Guide to create a Kubernetes cluster and delete ir when not need it.

gcloud container clusters create NAME
gcloud container clusters list
kubectl get nodes
...
gcloud container clusters delete NAME

Using Minikube requires to download/install minikube (i.e. use brew) then start Kubernetes executing:

minikube start
kubectl get nodes

This will start a VM with a single node Kubernetes cluster and the Docker engine.

Installing with kubeadm is another option, the instructions are here. Packages are available for Ubuntu and CentOS.

Run kubeadm init on the master node, that will return a token. On the worker node execute kubeadm join --token TOKEN MASTER-IP. This will create a network for IP-per-Pod criteria but you can use kubectl to apply a resource manifest of a different network. For example, a Weave network:

kubectl create -f https://git.io/weave-kube

Installing a Pod Network may be the next step, there are several to choose:

Calico: A flat layer 3 network which communicate without IP encapsulation. It's used in production, It's a simple and flexible networking model, scales well for large environments. It supports Network Policies
Canal: Is part of the Calico project. Allows integration with Flannel and implementation of Network Policies. It has the best of Calico and Flannel.
Flannel: A layer 3 IPv4 network between nodes. Focused on traffic between hosts, not containers, can use one of several backend mechanisms such as VXLAN. There is a flanneld agent on each node to allocate subnet leases for the host. It's easier to deploy it prior any Pod. While it's easier to deploy and supports a wide range of architectures, it does not support the use of Network Policies.
Kube-router: A single binary to "do it all". At this time it's in alpha stage.
Romana: aimed at network and security automation, large clusters, IPAM-aware topology and integration with kops
Weave Net: Add-on for CNI-enabled clusters.

CNI: Container Network Interface, which is a CNCF project. Several containers runtime use CNI.

The selection of the Pod Network for CNI depends of the expected demand on the cluster. There can be only one Pod Network per cluster (CNI-Genie is working on change this). The network must allow container-to-container, pod-to-pod, pod-to-service, and external-to-service communications. Docker use host-private networking, so using docker0 and veth requires to be on that host to communicate.

Other tools to install Kubernetes:

kubespary : Ansible playbook to setup Kubernetes on various OS. Once known as kargo.
kops: creates a Kubernetes cluster on AWS, in beta for GKE and alpha for VMware
kube-aws: creates a Kubernetes cluster on AWS using Cloud Formation.
kubicorn: leverages the use of kubeadm to build a cluster.

The best way to learn how to install Kubernetes using the manual commands is the kelsey hightower walkthorugh and kubernetes the hard way.

The article Picking the Right Solution helps to choose the best option.

On Linux, Kubernetes components may be running as systemd unit files, or run via a kubelet running on the master node (i.e. kubeadm).

There are 4 main deployment configurations:

Single-node: All components on same server. Good for testing, learning and developing.
Single master node, multiple workers: It's an etcd instance running on the master node with the API, scheduler and controller manager.
Multiple master nodes with HA, multiple workers: Adds more durability to te cluster. There is a Load Balancer in front of the API server, the scheduler and the controller manager. The etcd can still be single mode.
HA etcd, HA master nodes, multiple workers: Most advance and resilient setup. The etcd runs as a true cluster, providing HA and runs on nodes separate from Kubernetes master nodes.

The use of Kubernetes Federation also offers HA.

For any configuration, it's required some components as systemd. This is an example for controller-manager:

- name: kube-controller-manager.service
    command: start
    content: |
      [Unit]
      Description=Kubernetes Controller Manager
      Documentation=https://github.com/kubernetes/kubernetes
      Requires=kube-apiserver.service
      After=kube-apiserver.service
      [Service]
      ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-controller-manager -z /opt/bin/kube-controller-manager https://storage.googleapis.com/kubernetes-release/release/v1.7.6/bin/linux/amd64/kube-controller-manager
      ExecStartPre=/usr/bin/systemd +x /opt/bin/kube-controller-manager
      ExecStart=/opt/bin/kube-controller-manager \
      --service-account-private-key-file=/opt/bin/kube-serviceaccount.key \
      --root-ca-file=/var/run/kubernetes/apiserver.crt \
      --master=127.0.0.1:8080 \
      --logtostderr=true
      Restart=always
      RestartSec=10

This is not a perfect unit file, it downloads the controller binary and set the permissions to run. Every component is highly configurable.

Other option is to run the components as containers, this is what kubeadm does. There is a binary named hyperkube also available as a container image gcr.io/google_containers/hyperkube.

This method runs kubelet as a system daemon and configure it with a manifest specifying how to run the other components. The kubelet will manage them as pods, making sure they get restarted if they die.

To get help usage executing:

docker run --rm gcr.io/google_containers/hyperkube:v1.9.2 /hyperkube apiserver --help
docker run --rm gcr.io/google_containers/hyperkube:v1.9.2 /hyperkube scheduler --help
docker run --rm gcr.io/google_containers/hyperkube:v1.9.2 /hyperkube controller-manager --help

To compile Kubernetes from source, get the repository and build it with make. You need Go or Docker installed.

cd $GOPATH
git clone https://github.com/kubernetes/kubernetes
cd kubernetes
make

With Docker, execute make quick-release instead of the bare make above.

The _output/bin directory will contain all the built binaries.

Lab 3.1: Install Kubernetes

Create a Vagrantfile with the following content:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/bionic64"
  config.vm.network "forwarded_port", guest: 6443, host: 6443
  config.vm.hostname = "master"
  config.vm.network "private_network", ip: "10.0.0.10"
  config.vm.synced_folder "./shared_folder", "/home/vagrant/shared_folder"
  config.vm.provision "shell", inline: <<-SHELL
    # add the provisioning script here
  SHELL
end

The forwarded_port is to allow the access from the guest host, your computer, to the cluster using kubectl. The master IP is static and private, so there is no access to the node from outside the network. We used the shared folder to store the kubeconfig file, required to access the cluster with kubectl.

The vagrant provisioning script on the master node is about installing all the dependencies and install Kubernetes by kubeadm.

Install dependencies script:

# Upgrading system and installing dependencies
apt-get update && apt-get install -y apt-transport-https curl
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
apt-get update && apt-get upgrade -y

# Installing Docker, kubeadm, kubelet, kubectl and Kubernetes CNI
apt-get update
apt-get install -y \
  docker.io \
  kubeadm=1.11.1-00 \
  kubelet=1.11.1-00 \
  kubectl=1.11.1-00 \
  kubernetes-cni=0.6.0-00
apt-mark hold docker.io kubelet kubeadm kubectl kubernetes-cni
swapoff -a
systemctl enable docker.service

The script is going to install the latest supported version of Docker, and the version 1.11 of kubeadm, kubelet and kubectl, the Kubernetes CNI version that works with them is 0.6.0-00. It's important to use these versions, otherwise select the version of OS and Calico manifest that works with them (and maybe, modify the instructions)

Script to install Kubernetes by kubeadmin:

# Installing Kubernetes by kubeadmin
kubeadm init \
  --pod-network-cidr 192.168.0.0/16 \
  --apiserver-advertise-address=10.0.0.10 \
  --apiserver-cert-extra-sans=localhost

# Installing Canal
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/rbac.yaml --kubeconfig /etc/kubernetes/admin.conf
kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/canal.yaml --kubeconfig /etc/kubernetes/admin.conf

# Only in non-production environments:
kubectl taint nodes --all node-role.kubernetes.io/master- --kubeconfig /etc/kubernetes/admin.conf

# Share kubeconfig and copy it for user 'vagrant'
cp /etc/kubernetes/admin.conf /home/vagrant/shared_folder/config
sed 's/10.0.0.10/localhost/' /etc/kubernetes/admin.conf > /home/vagrant/shared_folder/remote_config
mkdir -p /home/vagrant/.kube
cp /home/vagrant/shared_folder/config /home/vagrant/.kube/
chown vagrant:vagrant -R /home/vagrant/.kube

The command kubeadm init will install and configure Kubernetes, it's important to use the network CIDR 192.168.0.0/16 because it's the same used by Calico. Also localhost is used to generate the certificates to make it possible to access the cluster API from the guest machine. The remote_config file is the same as the original kubeconfig just replacing the master IP for localhost.

This solution install Canal but it could also install Calico if you replace those 2 lines for:

kubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml

This uses version 2.6 because latest versions (v3.1 & v3.2) were failing.

Insert the above provisioning script and execute vagrant up. The Vagrantfile will download and create an Ubuntu VM, then will install Kubernetes.

Using the file shared_folder/remote_config as kubeconfig file, execute the following commands to validate the cluster:

export KUBECONFIG=shared_folder/remote_config
kubectl get nodes
node_name=$(kubectl get node -o jsonpath='{.items[*].status.addresses[?(@.type=="Hostname")].address}')
kubectl describe node $node_name
kubectl describe node $node_name | grep -i taint
kubectl get pods --all-namespaces

There should be one node in the cluster (master) and all the pods should be running.

The solution to this lab is in the Vagrantfile ./Lab_03/Vagrantfile.v1

Lab 3.2 Grow the Cluster

Execute vagrant destroy to destroy the VM's and the cluster.

Modify the Vagrantfile to define the master and the worker nodes. In the master provisioning script, get the node ip address, kubeadm token and token hash to save them in the shared folder:

token=$(kubeadm token list | tail -2 | head -1 | cut -f1 -d' ')
token_ca_cert=$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //')

echo $token > /home/vagrant/shared_folder/token
echo $token_ca_cert > /home/vagrant/shared_folder/token_ca_cert.hash

The token and the token certificate hash are required by kubeadm join to join the worker to the cluster. Only the master node knows these 2, so they are saved into the shared directory to be accessible by the worker(s).

Also, assign to kubelet the correct node IP address and restart the daemon:

echo "KUBELET_EXTRA_ARGS=--node-ip='${IP}'" >/etc/default/kubelet
systemctl daemon-reload
systemctl restart kubelet

This workaround is to fix a kubeadm issue/feature #203, it assign to the node the first IP found. This first IP is the same on every vagrant node (10.0.2.15), therefore calico failed to start up because both nodes have the same IP address.

Now for the worker provisioning repeat the same instructions as in the master and the following script:

token=$(cat /home/vagrant/shared_folder/token)
token_ca_cert=$(cat /home/vagrant/shared_folder/token_ca_cert.hash)

# Adding Worker to Master running on https://10.0.0.10:6443
kubeadm join \
  --token $token \
  --discovery-token-ca-cert-hash sha256:$token_ca_cert \
  10.0.0.10:6443

The final and easily customizable Vagrantfile is located at Solutions/Lab_03/Vagrantfile.

To create the cluster execute vagrant up or you may startup the cluster one node at a time with:

vagrant up master
vagrant up worker

Verify the cluster health executing the same instructions as for the previous lab (Lab 3.1)

Some of the pages used to solve this Lab are:

Another Kubernetes the hard way: https://github.com/kinvolk/kubernetes-the-hard-way-vagrant
Kubeadm playground: https://github.com/kubernetes/kubeadm/tree/master/vagrant
Another playground: https://github.com/davidkbainbridge/k8s-playground
Kubeadm issue #203: kubernetes/kubeadm#203

To destroy the cluster, execute:

vagrant destroy

And to cleanup everything, execute:

rm shared_folder/*
rm *.log
rm -rf .vagrant/
vagrant box remove ubuntu/bionic64 --all

The solution to this lab is in the Vagrantfile Lab_03/Vagrantfile.v2.

Lab 3.3: Deploy A Simple Application

A deployment is an Kubernetes object that deploy and monitor an application in a container.

Execute the following commands to deploy a nginx web server:

kubectl run nginx --image nginx
kubectl get deployments
kubectl describe deployment nginx
kubectl get events
kubectl get deployment nginx -o yaml
kubectl get deployment nginx -o yaml > nginx.yaml
kubectl delete deployment nginx

Edit nginx.yaml to remove the creationTimestamp, resourceVersion, selfLink, uid and status: lines.

Deploy again the nginx but this time using a manifest file with the deployment object:

kubectl create -f nginx.yaml
kubectl get deployment nginx -o yaml > temp_nginx.yaml
diff nginx.yaml temp_nginx.yaml

Another way to generate the manifest without pre-deploying it is:

kubectl run nginx --image=nginx --dry-run -o yaml
kubectl get deployments

And one more, from an existing deployed object but this time there is no need to edit the file:

kubectl get deployments nginx --export -o yaml

You can get the manifest file in JSON format using -o json.

The container in the deployed pods are running but are not accessible from outside the cluster. To view the nginx default web page, you need to create a service using either the command expose or modifying the manifest file. Executing kubectl expose deployment/nginx will throw an error because the --port flag is not used. So we either modify the manifest file adding the 3 lines to define the ports at the image: section, and then use the replace command to apply the change, like this:

containers:
- image: nginx
  imagePullPolicy: Always
  name: nginx
  ports:
  - containerPort: 80
    protocol: TCP

Then: kubectl replace -f nginx.yaml

Here we used the replace command because the container was create with the create sub-command. The best way from the very beginning was to use the sub-command apply so this is the only command to use when we have to update the deployment. Like this: kubectl apply -f nginx.yaml

To verify the change, execute:

kubectl get deploy,pod
kubectl get svc nginx   # or: kubectl get services nginx
kubectl get ep nginx    # or: kubectl get endpoints nginx

Running the expose subcommand now won't throw any error. And, notice that the Cluster IP from the service is not the node IP neither the pod endpoint IP

To know in which node the pod is running on use one of the following commands:

kubectl get pods -o wide | grep nginx
kubectl get pods -o jsonpath='{.items[*].spec.nodeName}'
kubectl describe pod nginx-UUID | grep 'Node:'

~~And to watch the traffic going back and forward on that container, login into the node where the pod is running to use the command tcpdump on the tun10 interface: sudo tcpdump -i tunl0~~

Now use curl to access the pod endpoint ~~and see all the messages going back and forth~~, but as the endpoint IP of the pod is not accessible from the guest computer, the curl has to be executed from the node where it's running.

ip=$(kubectl get ep nginx -o jsonpath='{.subsets[0].addresses[0].ip}')
vagrant ssh worker -c "curl http://${ip}:80"

To scale up the number of nginx servers running, increase the number of replicas of the deployment with:

kubectl get deployment nginx
kubectl scale deployment nginx --replicas=3
kubectl get deployment nginx
kubectl get pods

And the number of endpoint addresses have also increased, one per replica:

kubectl get endpoints nginx
ip_addresses=$(kubectl get ep nginx -o jsonpath='{.subsets[0].addresses[*].ip}')
for ip in ip_addresses; do
  echo "getting http://$ip:80"
  vagrant ssh worker -c "curl http://${ip}:80"
done

Using the endpoint address is not good nor practical as they will change every time the pod is restored. List the pods and delete one of them, then watch how a new one takes its place and the endpoint change:

kubectl get pods -o wide
pod=$(kubectl get po -o wide | tail -1 | awk '{print $1}')
kubectl delete pods ${pod}
kubectl get pods -o wide
kubectl get endpoints nginx

There is no need to get the new pod IP address to access the nginx web page, using the service IP or, in this case: Cluster IP, is enough and better. But, same as the endpoints, the Cluster IP is only accessible from within the cluster so use the curl command from any node of the cluster, i.e. the master node.

kubectl get services nginx
ip=$(kubectl get services nginx -o jsonpath='{.spec.clusterIP}')
port=$(kubectl get services nginx -o jsonpath='{.spec.ports[0].port}')
vagrant ssh master -c "curl http://${ip}:${port}"

To pause and resume, use the commands vagrant halt and vagrant up. Or, to destroy and partially cleanup execute:

vagrant destroy
rm *.yaml
rm *.log
rm shared_folder/*

The solution to this lab is in the manifest files nginx.yaml and nginx_service.yaml.

Lab 3.4: Access from Outside the Cluster

To print the environment variables in a pod execute the following commands to identify the pod name and execute a command inside it:

kubectl get pods
pod=$(kubectl get po -o wide | tail -1 | awk '{print $1}')
kubectl exec ${pod} -- env | grep KUBERNETES

Any command executed after the double dashes -- is executed inside the container. Try it with other commands such as ls -al.

Recreate the service but this time using LoadBalancer type:

kubectl get services
kubectl delete svc nginx
kubectl expose deployment nginx --type=LoadBalancer

Notice that LoadBalancer is only supported on some platforms like GCP, AWS, Azure and OpenStack. In this case, we are using Vagrant with VirtualBox, so the service should be exposed using the NodePort type.

kubectl delete svc nginx
kubectl expose deployment nginx --type=NodePort

Still it's not accessible from your computer because we just cannot access vagrant VM internal IP address, the way to access the service is forwarding the port. First, identify the port the service is using at the node (nodePort), then update the Vagrantfile to forward the host port 8080 to the service port.

$ kubectl get svc nginx -o jsonpath='{.spec.ports[0].nodePort}'
30001

And this inside the worker definition to forward any access to localhost:8080 into the worker node on port 30001:

w.vm.network :forwarded_port, host: "8080", guest: "30001"

Finally, reload vagrant to apply the changes to the VM's:

vagrant reload

Or, automate all this process adding the following lines in the worker definition:

ports.each { |host, guest|
  w.vm.network :forwarded_port, host: host, guest: guest
}

Add this ruby code:

env_ports=ENV["PORTS"]
env_ports="" if env_ports == nil

ports = Hash.new
env_ports.split(':').each do |pair|
  p=pair.split('>')

  if p.length == 1
    guest = p[0]
    host = (MIN_SERVICE_GUEST_PORT + guest.to_i - MIN_SERVICE_NODEPORT).to_s
  else
    host = p[0]
    guest = p[1]
  end

  ports[host] = guest
end

Then identify the port, define which port to map in the host machine (your computer) and pass it to the vagrant reload, like this:

port=$(kubectl get svc nginx -o jsonpath='{.spec.ports[0].nodePort}')
PORTS="8080>${port}" vagrant reload

The environment variable PORTS has the following format: "[host_port>]guest_port[:[host_port>guest_port]]", if the host_port is not specified, will use 3000 + (guest_port - 30000) for example:

"8080>30001:8090>30123" = {

localhost:8080 => worker:30001

localhost:8090 => worker:30123

}
"8080>30456:30123" = {

localhost:8080 => worker:30456

localhost:3123 => worker:30123

}

Optionally, export the service manifest to a file, so it always uses the same port (i.e. 30001) next time it's applied. Execute:

kubectl get service nginx -o yaml > nginx_service_nodeport.yaml

And edit the line with the parameter nodePort, like this:

ports:
- nodePort: 30001
  port: 80
  protocol: TCP
  targetPort: 80

To verify the access to the nginx service, check the service is scaling correctly, first reduce the number of replicas to zero, confirm that there is no resources, scale to two replicas and check the pods and the service.

kubectl scale deployment nginx --replicas=0
kubectl get pods
curl http://localhost:8080

kubectl scale deployment nginx --replicas=2
kubectl get pods -w
curl http://localhost:8080

If the latest curl fail, try again, it takes some time to get it working.

Also, try http://localhost:8080 using your preferred browser.

Cleanup everything to restore the cluster to the original state:

kubectl delete deployment nginx
kubectl delete ep nginx
kubectl delete service nginx

Notice that deleting the deployment does not delete the endpoint neither the service.

For the solution of this lab, were used the following pages:

Services: https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/
NodePort Service: https://kubernetes.io/docs/concepts/services-networking/service/#nodeport

The solution to the lab 3.4 is in the Vagrantfile Lab_03/Vagrantfile.v3 and the manifest file nginx_service_nodeport.yaml.

Wrapping up

To reproduce the entire lab, execute:

vagrant up
export KUBECONFIG=shared_folder/remote_config
kubectl get nodes -w
# wait until you see all the nodes ready
watch -n 0.5 kubectl get pods --all-namespaces
# wait until you see all the pods running
kubectl apply -f nginx.yaml
kubectl apply -f nginx_service_nodeport.yaml
kubectl get pods
kubectl get services
port=$(kubectl get svc nginx -o jsonpath='{.spec.ports[0].nodePort}')
PORTS="8080>${port}" vagrant reload
curl http://localhost:8080
# Only if you are on OSX, otherwise open a browser
open http://localhost:8080

# Cleanup:
kubectl delete deployment nginx
kubectl delete ep nginx
kubectl delete service nginx
vagrant destroy --force
# Deep cleanup:
rm shared_folder/*
rm *.log
rm -rf .vagrant/
# Deeper cleanup
vagrant box remove ubuntu/bionic64 --all

All the code is in the Vagrantfile. The manifest files were obtained exporting the deployment and service, with some minor modifications.

A similar Vagrantfile with more features, is in the GitHub project johandry/KoV.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ch03.md

Ch03.md

Chapter 3. Installation and Configuration

Lab 3.1: Install Kubernetes

Lab 3.2 Grow the Cluster

Lab 3.3: Deploy A Simple Application

Lab 3.4: Access from Outside the Cluster

Wrapping up

Files

Ch03.md

Latest commit

History

Ch03.md

File metadata and controls

Chapter 3. Installation and Configuration

Lab 3.1: Install Kubernetes

Lab 3.2 Grow the Cluster

Lab 3.3: Deploy A Simple Application

Lab 3.4: Access from Outside the Cluster

Wrapping up