This proposal attempts to improve the existing local cluster experience for kubernetes. The current local cluster experience is sub-par and often not functional. There are several options to setup a local cluster (docker, vagrant, linux processes, etc) and we do not test any of them continuously. Here are some highlighted issues:
- Docker based solution breaks with docker upgrades, does not support DNS, and many kubelet features are not functional yet inside a container.
- Vagrant based solution are too heavy and have mostly failed on OS X.
- Local linux cluster is poorly documented and is undiscoverable. From an end user perspective, they want to run a kubernetes cluster. They care less about how a cluster is setup locally and more about what they can do with a functional cluster.
From a high level the goal is to make it easy for a new user to run a Kubernetes cluster and play with curated examples that require least amount of knowledge about Kubernetes. These examples will only use kubectl and only a subset of Kubernetes features that are available will be exposed.
- Works across multiple OSes - OS X, Linux and Windows primarily.
- Single command setup and teardown UX.
- Unified UX across OSes
- Minimal dependencies on third party software.
- Minimal resource overhead.
- Eliminate any other alternatives to local cluster deployment.
- Enable developers to use the local cluster for kubernetes development.
- Simplifying kubernetes production deployment experience. Kube-deploy is attempting to tackle this problem.
- Supporting all possible deployment configurations of Kubernetes like various types of storage, networking, etc.
- Includes all the master components & DNS (Apiserver, scheduler, controller manager, etcd and kube dns)
- Basic auth
- Service accounts should be setup
- Kubectl should be auto-configured to use the local cluster
- Tested & maintained as part of Kubernetes core
Following are some of the existing solutions that attempt to simplify local cluster deployments.
Spread's UX is great! It is adapted from monokube and includes DNS as well. It satisfies almost all the requirements, excepting that of requiring docker to be pre-installed. It has a loose dependency on docker. New releases of docker might break this setup.
Kmachine is adapted from docker-machine. It exposes the entire docker-machine CLI. It is possible to repurpose Kmachine to meet all our requirements.
Single binary that runs all kube master components. Does not include DNS. This is only a part of the overall local cluster solution.
The kube-up.sh script included in Kubernetes release supports a few Vagrant based local cluster deployments. kube-up.sh is not user friendly. It typically takes a long time for the cluster to be set up using vagrant and often times is unsuccessful on OS X. The Core OS single machine guide uses Vagrant as well and it just works. Since we are targeting a single command install/teardown experience, vagrant needs to be an implementation detail and not be exposed to our users.
To avoid exposing users to third party software and external dependencies, we will build a toolbox that will be shipped with all the dependencies including all kubernetes components, hypervisor, base image, kubectl, etc. Note: Docker provides a similar toolbox. This "Localkube" tool will be referred to as "Minikube" in this proposal to avoid ambiguity against Spread's existing "localkube". The final name of this tool is TBD. Suggestions are welcome!
Minikube will provide a unified CLI to interact with the local cluster. The CLI will support only a few operations: - Start - creates & starts a local cluster along with setting up kubectl & networking (if necessary) - Stop - suspends the local cluster & preserves cluster state - Delete - deletes the local cluster completely - Upgrade - upgrades internal components to the latest available version (upgrades are not guaranteed to preserve cluster state)
For running and managing the kubernetes components themselves, we can re-use Spread's localkube. Localkube is a self-contained go binary that includes all the master components including DNS and runs them using multiple go threads. Each Kubernetes release will include a localkube binary that has been tested exhaustively.
To support Windows and OS X, minikube will use libmachine internally to create and destroy virtual machines. Minikube will be shipped with an hypervisor (virtualbox) in the case of OS X. Minikube will include a base image that will be well tested.
In the case of Linux, since the cluster can be run locally, we ideally want to avoid setting up a VM. Since docker is the only fully supported runtime as of Kubernetes v1.2, we can initially use docker to run and manage localkube. There is risk of being incompatible with the existing version of docker. By using a VM, we can avoid such incompatibility issues though. Feedback from the community will be helpful here.
If the goal is to run outside of a VM, we can have minikube prompt the user if docker is unavailable or version is incompatible. Alternatives to docker for running the localkube core includes using rkt, setting up systemd services, or a System V Init script depending on the distro.
To summarize the pipeline is as follows:
minikube -> libmachine -> virtualbox/hyper V -> linux VM -> localkube
minikube -> docker -> localkube
- Kubernetes users will probably already have it
- No extra work for us
- Only one VM/daemon, we can just reuse the existing one
- Not designed to be wrapped, may be unstable
- Might make configuring networking difficult on OS X and Windows
- Versioning and updates will be challenging. We can mitigate some of this with testing at HEAD, but we'll - inevitably hit situations where it's infeasible to work with multiple versions of docker.
- There are lots of different ways to install docker, networking might be challenging if we try to support many paths.
- We control the entire experience
- Networking might be easier to build
- Docker can't break us since we'll include a pinned version of Docker
- Easier to support rkt or hyper in the future
- Would let us run some things outside of containers (kubelet, maybe ingress/load balancers)
- More work
- Extra resources (if the user is also running docker-machine)
- Confusing if there are two docker daemons (images built in one can't be run in another)
- Always needs a VM, even on Linux
- Requires installing and possibly understanding Vagrant.
- Minikube will be released independent of Kubernetes core in order to facilitate fixing of issues that are outside of Kubernetes core.
- The latest version of Minikube is guaranteed to support the latest release of Kubernetes, including documentation.
- The Google Cloud SDK will package minikube and provide utilities for configuring kubectl to use it, but will not in any other way wrap minikube.