Skip to content

Commit

Permalink
Updating Exteding Docker with PVC case (#862)
Browse files Browse the repository at this point in the history
  • Loading branch information
@jayasimha-raghavan-unskript
jayasimha-raghavan-unskript authored Sep 9, 2023
1 parent fb3b841 commit 2075545
Show file tree
Hide file tree
Showing 4 changed files with 120 additions and 67 deletions.
3 changes: 2 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ cd Awesome-CloudOps-Automation

```
docker run -it -p 8888:8888 \
-v $HOME/Awesome-CloudOps-Automation/custom:/unskript/data \
-v $HOME/Awesome-CloudOps-Automation/actions:/unskript/data/actions \
-v $HOME/Awesome-CloudOps-Automation/runbooks:/unskript/data/runbooks \
-v $HOME/.unskript:/unskript/credentials \
-e ACA_AWESOME_MODE=1 \
-e OPENAI_ORGANIZATION_ID=<your openAI org> \
Expand Down
127 changes: 64 additions & 63 deletions README_extending_docker.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,72 +7,49 @@


## Extending the docker
You can use our base docker to extend the functionality to fit your need. The steps below could be used to package your custom Actions and re-build your custom docker that you can upload and distribute to/from any docker registry.
You can use our base docker to extend the functionality to fit your need. The steps below could be used to package your custom Actions/Runbooks and re-build your custom docker that you can upload and distribute to/from any docker registry.

### Pre-requisite
1. You have a working Python 3 environment installed on your build system
2. You have Docker-ce installed on your build system

### Steps
1. If you have not already done, please check-out the Awesome-CloudOps-Automation repo with this command

## Pre-requisites
1. You are submoduling our Awesome-CloudOps-Automation to your existing
Git repo.
```
git clone https://github.com/unskript/Awesome-CloudOps-Automation
export ACA_ROOTDIR=$PWD/Awesome-CloudOps-Automation
```

2. Lets assume that all your development will be done in a directory called `custom` under the Awesome-CloudOps-Automation directory. You could set up `custom` using any of the following methods.
1. Create directory under Awesome-CloudOps-Automation by name `custom`
```
cd $ACA_ROOTDIR
mkdir custom
```
2. Submodule your Git repo that has your custom Actions in it.
```
cd $ACA_ROOTDIR
git submodule add https://<YOUR REPO LOCATION> custom
```
3. Next, lets setup an environment variable to the custom directory. Set an environment variable to point to the name of the custom directory.
cd $YOUR_REPO_DIRECTORY
git submodule add http://github.com/Awesome-CloudOps-Automation.git Awesome-CloudOps-Automation
```
export ACA_CUSTOM_DIR_NAME=$ACA_ROOTDIR/custom
2. In same directory, you will need to create two sub-folders.
```
mkdir -p $YOUR_REPO_DIRECTORY/runbooks $YOUR_REPO_DIRECTORY/actions
```

4. Launch the Awesome CloudOps Docker.
```
docker run -it -p 8888:8888 \
-v $HOME/.unskript:/unskript \
-v $ACA_CUSTOM_DIR_NAME:/data \
-e ACA_AWESOME_MODE=1 \
--user root \
unskript/awesome-runbooks:latest
```
> Tip: If you are interested in building your custom docker image off of a Tag, you can replace the `latest` keyword
> in the above command with the tag number. You can find the tags [Here](https://hub.docker.com/r/unskript/awesome-runbooks/tags)
* Here you may notice we have two `-v` mount point. The first one `$HOME/.unskript` is for storing credentials.
* The second mount point `$ACA_CUSTOM_DIR_NAME` is where we save custom Actions or custom Runbooks.
> Note: This means any content that is created will survive Docker reboots.
* You would see a Welcome Message once the Docker starts. At this juncture point your browser to `http://127.0.0.1:8888/lab/tree/GetStarted.ipynb` (We recommend Google Chrome or MS Edge or Chromium)
5. Once the page is loaded, Search for any pre-built actions by typing keywords like `aws`, `kubernetes` `kubectl`, `postgresql`, `mongo` etc. To pick the Action that you want to extend, drag-n-drop it to the main notebook area. You can refer [to this](https://docs.unskript.com) to get familiar with the UI.
6. After you are done with the modifications, you can use the `Save-As` option to save your custom Action. You can refer [to this](https://docs.unskript.com) on how to save custom Action.
3. The Directory structure resulting would be something like this
```
YOUR_REPO_DIRECTORY/
actions/
runbooks/
Awesome-CloudOps-Auatomation/
your-repo-folders/
your-repo-files
...
```
4. You have a working Python 3 environment installed on your build system
5. You have `make` and other build tools installed on your bulid system
6. You have Docker-ce installed and working on your build system

> Tip: If you want to verify the modification, you can create a credential for the given connector and test your modification to make sure
> you are satisfied with the changes.

7. Next step is to build your custom docker from the Awesome-CloudOps-Automation directory
## Building Custom Docker
1. To build your custom docker. You need to set two environment variables
`CUSTOM_DOCKER_NAME` and `CUSTOM_DOCKER_VERSION`. If not set, by default the
Make rule will assume `my-custom-docker` and `0.1.0` as values for these
variables.

```
export CUSTOM_DOCKER_NAME=my-awesome-docker
export CUSTOM_DOCKER_VERSION='0.1.0'
cp ./build/templates/Dockerfile.template Dockerfile
docker build -t $CUSTOM_DOCKER_NAME:$CUSTOM_DOCKER_VERSION .
cd $YOUR_REPO_DIRECTORY
cp Awesome-CloudOps-Automation/build/templates/Dockerfile.template Dockerfile
cp Awesome-CloudOps-Automation/build/templates/Makefile.extend-docker.template Makefile
make -f Makefile build
```

It may take a few minutes to build the docker, once built, you can verify it using
Expand All @@ -82,15 +59,39 @@ You can use our base docker to extend the functionality to fit your need. The st
$CUSTOM_DOCKER_NAME:$CUSTOM_DOCKER_VERSION
```

This would run your `custom docker` and you can point your browser to `http://127.0.0.1:8888/lab/tree/Welcome.ipynb`!
Audit logs will not be persisted across docker deletion. In order to persist them, you can use the following command (assuming the data needs to be stored in $HOME/data)
This would run your `custom docker` and you can point your browser to `http://127.0.0.1:8888/awesome`!

```
docker run -it -p 8888:8888 \
-v $HOME/data:/data \
$CUSTOM_DOCKER_NAME:$CUSTOM_DOCKER_VERSION
```
8. Push your `custom docker` to any docker registry for redistribution.
2. Push your `custom docker` to any docker registry for redistribution.
<br/>



## How to Copy Custom Actions and Runbook

If you have deployed our Awesome runbook as a Kubernetes POD then follow the step below
1. Copy the custom actions from the POD to your local machine so you can bundle into your custom Docker for re-distribution
```
kubectl cp <AWESOME_POD_NAME>:/unskript/data/runbooks -n <NAMESPACE> $YOUR_REPO_DIRECTORY/runbooks
kubectl cp <AWESOME_POD_NAME>:/unskript/data/actions -n <NAMESPACE> $YOUR_REPO_DIRECTORY/actions
cd $YOUR_REPO_DIRECTORY
git clone https://github.com/unskript/Awesome-CloudOps-Automation.git
Example:
kubectl cp awesome-runbooks-0:/unskript/data/actions -n awesome-ops $YOUR_REPO_DIRECTORY/actions
kubectl cp awesome-runbooks-0:/unskript/data/runbooks -n awesome-ops $YOUR_REPO_DIRECTORY/runbooks
```

If you have deployed our Awesome runbook as a Docker instance, then you can use
the following step.
```
export CONTAINER_ID=`docker ps | grep awesome-runbooks | awk '{print $1}'`
docker cp $CONTAINER_ID:/unskript/data/actions $HOME/Workspace/acme/actions
docker cp $CONTAINER_ID:/unskript/data/runbooks $HOME/Workspace/acme/runbooks
```

Lets consider two scenarios as starting point for extending Awesome docker

### How to create Custom Actions

You can refer to [this link](https://docs.unskript.com/unskript-product-documentation/actions/create-custom-actions) on how to create custom Action
6 changes: 3 additions & 3 deletions build/templates/Dockerfile.template
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
FROM unskript/awesome-runbooks:latest as base
RUN mkdir -p /data
RUN mkdir -p /unskript
ADD custom /data/custom
COPY custom/actions /unskript/data
COPY custom/runbooks /unskript/data

CMD ["./start.sh"]

51 changes: 51 additions & 0 deletions build/templates/Makefile.extend-docker.template
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#
# Copyright (c) 2023 unSkript.com
# All rights reserved.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE
#
#
.PHONY: bulid

# ACTIONS DIRECTORY
ACTION_DIRECTORY = actions

# RUNBOOKS DIRECTORY
RUNBOOK_DIRECTORY = runbooks

# RUNBOOKS DIRECTORY
AWESOME_DIRECTORY = Awesome-CloudOps-Automation

# CUSTOM DIRECTORY
CUSTOM_DIRECTORY = custom

# Set Default values
CUSTOM_DOCKER_NAME ?= my-awesome-docker
CUSTOM_DOCKER_VERSION ?= 0.1.0

pre-build:
@echo "Preparing To create custom Docker build"
if [ ! -d "$(ACTION_DIRECTORY)" ]; then\
echo "Actions Directory does not exist, It is needed to build the custom docker image"; \
exit 1; \
fi
if [ ! -d "$(RUNBOOK_DIRECTORY)" ]; then\
echo "Runbooks Directory does not exist, It is needed to build the custom docker image"; \
exit 1; \
fi
if [ ! -d "$(AWESOME_DIRECTORY)" ]; then\
echo "Awesome-CloudOps-Automation Directory does not exist, It is needed to build the custom docker image"; \
exit 1; \
fi
@echo "Verified all pre-requisites are met, proceeding to build custom docker image"
@mkdir -p $(CUSTOM_DIRECTORY)
@cp -Rf $(ACTION_DIRECTORY) $(CUSTOM_DIRECTORY)/actions
@cp -Rf $(RUNBOOK_DIRECTORY) $(CUSTOM_DIRECTORY)/runbooks
@cp $(AWESOME_DIRECTORY)/build/templates/Dockerfile.template Dockerfile


build: pre-build
@echo "Using \n Custom Docker Name: $(CUSTOM_DOCKER_NAME) Custom Docker Version: $(CUSTOM_DOCKER_VERSION)"
@docker build -t $(CUSTOM_DOCKER_NAME):$(CUSTOM_DOCKER_VERSION) -f Dockerfile .

0 comments on commit 2075545

Please sign in to comment.