The Plex Sandbox Operator is an operator for Kubernetes that enables authenticated users to create their own isolated environments.
This repository contains a deploy folder which contains all of the manifests required to deploy the operator, as well as a kustomization.yaml file.
If you would like to apply your own customizations, reference the deploy folder and the version in your kustomization.yaml.
(version used in kubectl apply -k .)
bases:
- git::https://github.com/plexsystems/sandbox-operator.git//deploy?ref=v0.10.1Latest version of Kustomize if installed as a standalone. Also version embedded in flux.
resources:
- git::https://github.com/plexsystems/sandbox-operator.git//deploy?ref=v0.10.1The example folder shows one example of how to customize the operator.
A bundle.yaml is provided in the root of the repository which can then be applied via kubectl apply.
A ClusterRole resource and a ClusterRoleBinding resource will be created to enable authenticated users to create Sandbox resources.
| Verbs | API Groups | Resources |
|---|---|---|
| create, list, get | operators.plex.dev | sandboxes |
| API Group | Name | Subjects |
|---|---|---|
| rbac.authorization.k8s.io | sandbox-users | system:authenticated |
A CustomResourceDefinition named Sandbox will be created.
An example manifest for the Sandbox CRD is as follows:
apiVersion: operators.plex.dev/v1alpha1
kind: Sandbox
metadata:
name: test
spec:
size: small
owners:
- [email protected]The Sandbox operator can leverage different clients, depending upon how authenitcation is configured for your cluster.
If Azure credentials are provided to the operators environment, it will perform a lookup of each user in the owners field and fetch that users ObjectID inside of Azure using the Microsoft Graph API.
This enables users to create Sandboxes with friendly names in the owners field (such as the owners email address) and have the operator itself handle the mapping to the ObjectID when creating the Kubernetes resources.
To use the Azure client, include the following environment variables:
AZURE_CLIENT_IDAZURE_TENANT_IDAZURE_CLIENT_SECRET
Your Azure Service Principal will need the following Application permission for the Azure Active Directory Graph API (00000002-0000-0000-c000-000000000000):
Directory.Read.All (5778995a-e1bf-45b8-affa-663a9f3f4d04)
If no credentials are provided, the operator will create the Role and ClusterRole bindings using the values listed in the owners field.
By default, the operator will not create any secrets in the provisioned namespace.
If the PULL_SECRET_NAME environment variable is set, the operator will copy your clusters pull secret to the provisioned namespace and patch the default service account.
PULL_SECRET_NAME should be the name of the pull secret that exists in your cluster. By default, the operator will look for your secret in the default namespace.
To have the operator look in a different namespace for the pull secret, use the PULL_SECRET_NAMESPACE environment variable.
To create a Sandbox, apply a Sandbox CRD to the target cluster.
The following will create a Sandbox called foo (the resulting namespace being sandbox-foo), and assign the RBAC policies to user [email protected].
apiVersion: operators.plex.dev/v1alpha1
kind: Sandbox
metadata:
name: foo
spec:
size: small
owners:
- [email protected]$ kubectl apply -f foo.yaml
sandboxes.operators.plex.dev "foo" createdAssuming the name of the created Sandbox is named foo, the following resources will be created per Sandbox:
| Verbs | API Groups | Resources | ResourceNames |
|---|---|---|---|
| * | operators.plex.dev | sandboxes | sandbox-foo |
This is created so that only users defined in the owners field can delete their Sandboxes.
One ClusterRoleBinding per name in the owners field. Bindings are added and removed as users are added and removed from the owners field.
| Verbs | API Groups | Resources |
|---|---|---|
| * | core | pods, pods/log, pods/portforward, services, services/finalizers, endpoints, persistentvolumeclaims, events, configmaps, replicationcontrollers |
| create | core | secrets |
| * | apps, extensions | deployments, daemonsets, replicasets, statefulsets |
| * | autoscaling | horizontalpodautoscalers |
| * | batch | jobs, cronjobs |
| create, list, get | rbac.authorization.k8s.io | roles, rolebindings |
One RoleBinding per name in the owners field. Bindings are added and removed as users are added and removed from the owners field.
The ResourceQuota that is applied to the Namespace depends on the size of the Sandbox that was created. Defaults to small if no size is given.
| Resource Name | Quantity |
|---|---|
| ResourceRequestsCPU | 0.25 |
| ResourceLimitsCPU | 0.5 |
| ResourceRequestsMemory | 250Mi |
| ResourceLimitsMemory | 500Mi |
| ResourceRequestsStorage | 10Gi |
| ResourcePersistentVolumeClaims | 2 |
| Resource Name | Quantity |
|---|---|
| ResourceRequestsCPU | 1 |
| ResourceLimitsCPU | 2 |
| ResourceRequestsMemory | 2Gi |
| ResourceLimitsMemory | 8Gi |
| ResourceRequestsStorage | 40Gi |
| ResourcePersistentVolumeClaims | 8 |
After the Sandbox has been created, you can add or remove owners that are associated to it.
For example, to add [email protected] as an owner, add their name to the list of owners and apply the changes:
apiVersion: operators.plex.dev/v1alpha1
kind: Sandbox
metadata:
name: foo
spec:
size: small
owners:
- [email protected]
- [email protected]$ kubectl apply -f sandbox-foo.yaml
sandboxes.operators.plex.dev "foo" configuredThis will cause the operator to add ClusterRoleBinding and RoleBinding resources to match the owners list.
When owners are removed from the Sandbox, their ClusterRoleBinding and RoleBinding will also be removed.
To delete a Sandbox, delete the Sandbox resource from the cluster:
$ kubectl delete sandbox foo
sandboxes.operators.plex.dev "foo" deletedDeleting a Sandbox will delete the Namespace as well as the ClusterRole and ClusterRoleBinding resources.
The operator exposes two metric ports for the /metrics endpoint:
- Port
8383exposes metrics for the operator itself - Port
8686exposes metrics for theSandboxCRD
Additionally, if prometheus-operator is installed into the cluster, a ServiceMonitor is created for the operator.
No external tooling is required to develop and build the operator. However, some tooling is required to run the integration tests:
The provided Makefile contains commands that assist with running the tests for the operator.
make test-unit will use an in-memory kubernetes client to validate and test your changes without the need for an external Kubernetes cluster.
make test-integration will create a Kubernetes cluster for you, using Kind, and deploy the operator to it. The integration tests will then be ran against the newly created cluster.
To test the operator with different versions of Kubernetes, you can use the KUBERNETES_VERSION variable when calling make.
For example, to test on Kubernetes v1.16.3, run the following command:
make test-integration KUBERNETES_VERSION=v1.16.3
We ❤️ pull requests. If you have a question, feedback, or would like to contribute — please feel free to create an issue or open a pull request!