Note
Using this method you can only install resources having the the latest version of a CRD, as there are no conversion webhooks running to convert other versions. This is the reason you need to setup test certificates using the steps below too. Use this method only to try out the operator to get familiar.
-
Clone the repository into the following folder
<GOPATH>/src/github.com/Azure
. -
Make sure the environment variable
GO111MODULE
is set toon
.export GO111MODULE=on
-
Make sure all the below environment variables have been set appropriately. The possible values for
AZURE_CLOUD_ENV
are AzurePubliCloud (default if unspecified), AzureChinaCloud, AzureUSGovernmentCloud, AzureGermanCloud.export AZURE_TENANT_ID=xxxx export AZURE_SUBSCRIPTION_ID=xxxx export AZURE_CLIENT_ID=xxxx export AZURE_CLIENT_SECRET=xxxx export AZURE_CLOUD_ENV=AzurePublicCloud
-
Install test certificates using
make generate-test-certs
. -
Install the CRDs defined in the config/crd/bases folder using the command
make install
You will see output as below.kubectl apply -f config/crd/bases customresourcedefinition.apiextensions.k8s.io/eventhubnamespaces.azure.microsoft.com created customresourcedefinition.apiextensions.k8s.io/eventhubs.azure.microsoft.com created customresourcedefinition.apiextensions.k8s.io/resourcegroups.azure.microsoft.com created customresourcedefinition.apiextensions.k8s.io/azuresqldatabases.azure.microsoft.com created customresourcedefinition.apiextensions.k8s.io/azuresqlfirewallrules.azure.microsoft.com created customresourcedefinition.apiextensions.k8s.io/azuresqlservers.azure.microsoft.com configured
-
Copy the generated certificates to the correct location by first attempting to run the operator locally using
make run
This will cause the operator to run (briefly) and will display an error (as well the correct location to copy generated certificates to). You will see something similar to this in the terminal window indicating that the controller started running and then subsequently failed:go fmt ./... go vet ./... go run ./main.go ... 2020-05-13T17:04:26.587-0700 INFO controller-runtime.builder skip registering a validating webhook, admission.Validator interface is not implemented {"GVK": "azure.microsoft.com/v1alpha1, Kind=AzureSqlFirewallRule"} 2020-05-13T17:04:26.587-0700 INFO controller-runtime.builder conversion webhook enabled {"object": {"metadata":{"creationTimestamp":null},"spec":{"server":""},"status":{}}} 2020-05-13T17:04:26.587-0700 INFO controller-runtime.builder skip registering a mutating webhook, admission.Defaulter interface is not implemented {"GVK": "azure.microsoft.com/v1alpha1, Kind=AzureSqlFailoverGroup"} 2020-05-13T17:04:26.587-0700 INFO controller-runtime.builder skip registering a validating webhook, admission.Validator interface is not implemented {"GVK": "azure.microsoft.com/v1alpha1, Kind=AzureSqlFailoverGroup"} 2020-05-13T17:04:26.587-0700 INFO controller-runtime.builder conversion webhook enabled {"object": {"metadata":{"creationTimestamp":null},"spec":{"location":"","server":"","failoverpolicy":"","failovergraceperiod":0,"secondaryserver":"","secondaryserverresourcegroup":"","databaselist":null},"status":{}}} 2020-05-13T17:04:26.587-0700 INFO controller-runtime.builder skip registering a mutating webhook, admission.Defaulter interface is not implemented {"GVK": "azure.microsoft.com/v1alpha1, Kind=BlobContainer"} 2020-05-13T17:04:26.587-0700 INFO controller-runtime.builder skip registering a validating webhook, admission.Validator interface is not implemented {"GVK": "azure.microsoft.com/v1alpha1, Kind=BlobContainer"} 2020-05-13T17:04:26.587-0700 INFO controller-runtime.builder conversion webhook enabled {"object": {"metadata":{"creationTimestamp":null},"spec":{"location":""},"status":{}}} 2020-05-13T17:04:26.587-0700 INFO setup starting manager 2020-05-13T17:04:26.587-0700 INFO controller-runtime.webhook.webhooks starting webhook server 2020-05-13T17:04:26.587-0700 INFO controller-runtime.manager starting metrics server {"path": "/metrics"} 2020-05-13T17:04:26.588-0700 DEBUG controller-runtime.manager non-leader-election runnable finished {"runnable type": "*webhook.Server"} 2020-05-13T17:04:26.588-0700 INFO controller-runtime.controller Starting EventSource {"controller": "keyvault", "source": "kind source: /, Kind="} 2020-05-13T17:04:26.588-0700 ERROR setup problem running manager {"error": "open /var/folders/1_/d50yhxgd357cfl0yjdclppfc0000gn/T/k8s-webhook-server/serving-certs/tls.crt: no such file or directory"} github.com/go-logr/zapr.(*zapLogger).Error /Users/username/Code/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128 main.main /Users/username/Code/go/src/github.com/WilliamMortlMicrosoft/azure-service-operator/main.go:781 runtime.main /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/proc.go:203 exit status 1 make: *** [run] Error 1`
Look for the listed directory given in the problem running manager log entry, above that directory is: "/var/folders/1_/d50yhxgd357cfl0yjdclppfc0000gn/T/". Next, run the following command
cp -r /tmp/k8s-webhook-server /var/folders/1_/d50yhxgd357cfl0yjdclppfc0000gn/T/
.Please note, copying the generated certificates into the this folder only needs to be done only once (or when the machine is rebooted or the local Kubernetes cluster is cleared). You do not need to do this step every time.
-
Run the operator locally using
make run
This will cause the operator to run and "watch" for events on this terminal. You will need to open a new terminal to trigger the creation of a custom resource.You will see something like this on the terminal window indicating that the controller is running.
go fmt ./... go vet ./... go run ./main.go 2019-09-24T12:18:10.419-0600 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"} 2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "eventhub", "source": "kind source: /, Kind="} 2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "resourcegroup", "source": "kind source: /, Kind="} 2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "eventhubnamespace", "source": "kind source: /, Kind="} 2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a mutating webhook {"GVK": "azure.microsoft.com/v1, Kind=EventhubNamespace", "path": "/mutate-azure-microsoft-com-v1-eventhubnamespace"} 2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a validating webhook {"GVK": "azure.microsoft.com/v1, Kind=EventhubNamespace", "path": "/validate-azure-microsoft-com-v1-eventhubnamespace"} 2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a mutating webhook {"GVK": "azure.microsoft.com/v1, Kind=Eventhub", "path": "/mutate-azure-microsoft-com-v1-eventhub"} 2019-09-24T12:18:10.424-0600 INFO controller-runtime.builder Registering a validating webhook {"GVK": "azure.microsoft.com/v1, Kind=Eventhub", "path": "/validate-azure-microsoft-com-v1-eventhub"} 2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "azuresqlserver", "source": "kind source: /, Kind="} 2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "azuresqldatabase", "source": "kind source: /, Kind="} 2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "azuresqlfirewallrule", "source": "kind source: /, Kind="} 2019-09-24T12:18:10.424-0600 INFO setup starting manager 2019-09-24T12:18:10.424-0600 INFO controller-runtime.manager starting metrics server {"path": "/metrics"} 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "eventhub"} 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "resourcegroup"} 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "azuresqldatabase"} 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "azuresqlfirewallrule"} 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "eventhubnamespace"} 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "azuresqlserver"} 2019-09-24T12:18:10.527-0600 INFO controller-runtime.certwatcher Updated current TLS certiface 2019-09-24T12:18:10.527-0600 INFO controller-runtime.certwatcher Starting certificate watcher 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "azuresqldatabase", "worker count": 1} 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "resourcegroup", "worker count": 1} 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "azuresqlfirewallrule", "worker count": 1} 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "eventhub", "worker count": 1} 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "azuresqlserver", "worker count": 1} 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "eventhubnamespace", "worker count": 1}
-
Open a new terminal window. Trigger the creation of a custom resource using kubectl and the sample YAML file provided. For instance, you would use the following command to create a SQL server:
kubectl apply -f config/samples/azure_v1alpha1_azuresqlserver.yaml azuresqlserver.azure.microsoft.com/sqlserver-sample created
-
You should see logs on the other terminal from the operator when this custom resource is being created.
2019-09-24T12:27:12.450-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"AzureSqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1alpha1","resourceVersion":"194427"}, "reason": "Updated", "message": "finalizer azuresqlserver.finalizers.azure.com added"} 2019-09-24T12:27:12.450-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"AzureSqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1alpha1","resourceVersion":"194427"}, "reason": "Submitting", "message": "starting resource reconciliation"} 2019-09-24T12:27:17.129-0600 INFO controllers.AzureSqlServer mutating secret bundle 2019-09-24T12:27:17.144-0600 INFO controllers.AzureSqlServer waiting for provision to take effect {"azuresqlserver": "default/sqlserver-sample1"} 2019-09-24T12:27:17.350-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"AzureSqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1alpha1","resourceVersion":"194437"}, "reason": "Checking", "message": "instance in NotReady state"} 2019-09-24T12:27:17.359-0600 INFO controllers.AzureSqlServer Got ignorable error {"azuresqlserver": "default/sqlserver-sample1", "type": "ResourceNotFound"} 2019-09-24T12:27:17.564-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"AzureSqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1alpha1","resourceVersion":"194439"}, "reason": "Checking", "message": "instance in NotReady state"} 2019-09-24T12:27:17.570-0600 INFO controllers.AzureSqlServer Got ignorable error {"azuresqlserver": "default/sqlserver-sample1", "type": "ResourceNotFound"} 2019-09-24T12:28:17.805-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"AzureSqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1alpha1","resourceVersion":"194439"}, "reason": "Checking", "message": "instance in Ready state"} 2019-09-24T12:28:19.010-0600 DEBUG controller-runtime.controller Successfully Reconciled {"controller": "sqlserver", "request": "default/sqlserver-sample1"} 2019-09-24T12:28:19.010-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"AzureSqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1alpha1","resourceVersion":"194518"}, "reason": "Provisioned", "message": "sqlserver sqlserver-sample1 provisioned "} 2019-09-24T12:28:19.202-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"AzureSqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1alpha1","resourceVersion":"194518"}, "reason": "Checking", "message": "instance in Ready state"} 2019-09-24T12:28:20.331-0600 DEBUG controller-runtime.controller Successfully Reconciled {"controller": "azuresqlserver", "request": "default/sqlserver-sample1"} 2019-09-24T12:28:20.331-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"AzureSqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1alpha1","resourceVersion":"194518"}, "reason": "Provisioned", "message": "sqlserver sqlserver-sample1 provisioned "}
-
Once the operator is running locally, in order to view debugging (Prometheus-based) metrics for the Azure operator, open a web browser and navigate to the Metrics Endpoint.
If you are using the Kubernetes cluster that comes with "Docker Desktop" and would like to view the Prometheus metrics, you can redirect port 8080 to the local machine using the following command:
kubectl port-forward deployment/controller-manager 8080
Then, open a web browser and navigate to the Metrics Endpoint.
If you're using VSCode with Remote - Containers extensions installed, you can quickly have your environment set up and ready to go, with everything you need to get started.
- Clone the repository and open the folder in VS Code. (will refer to the folder as PROJECT_DIR)
- Create a new file at $PROJECT_DIR/.env with the following variables set for your environment:
AZURE_TENANT_ID=
AZURE_SUBSCRIPTION_ID=
AZURE_CLIENT_ID=
AZURE_CLIENT_SECRET=
- Click on the green button in the lower left-hand corner and select "Remote-Containers: Reopen in Container" from the drop-down
- VSCode will relaunch and start building our development container. This will install all the necessary dependencies required for you to begin developing.
- Once the container has finished building, you can now start testing your Azure Service Operator within your own local kubernetes environment via the terminal inside VSCode.
- Install test certificates using
make generate-test-certs
. - Install CRDs using
make install
. - Run the project using
make run
. - In another terminal, use
kubectl apply
with the sample YAML files to create custom resources for testing. For eg., usekubectl apply -f config/samples/azure_v1alpha1_azuresqlserver.yaml
from the terminal to create a SQL server using the operator.kubectl describe SqlServer
would show the events that indicate if the resource is created or being created.
If you are ready to contribute a new operator to the Azure Service Operators project, please follow the step-by-step guide here.
If you make changes to the Chart or if you make changes to the CRDs, you will need to update the Helm Chart and check-in the updated chart using the following steps:
- After you make the changes to the CRDs or the chart, run
make helm-chart-manifests
from the project root. - Add all the modified files generated including the .tgz file, to your commit.
- Include these files in your Pull Request.