diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index fd30c6810..3b0f90d28 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -22,8 +22,8 @@ jobs: uses: Tiryoh/actions-mkdocs@v0 with: mkdocs_version: 'latest' - requirements: 'docs/requirements.txt' - configfile: 'mkdocs.yml' + requirements: 'docs/requirements-github.txt' + configfile: 'mkdocs-github.yml' - name: show contents of compiled docs diff --git a/README.md b/README.md index bd4ece734..6c4c322d7 100644 --- a/README.md +++ b/README.md @@ -28,13 +28,21 @@ The rest of the information on this page is aimed at current and potential contr To install the development environment, run ```bash -bash script/install.bash +bash script/env.sh ``` +There is a script to download all the docker containers used in the project. You can download them using + +```bash +bash script/docker.sh +``` + +**CAVEAT**: The docker images are large and are likely to consume about 5GB of bandwidth and 15GB of space. You will have to download the docker images on a really good network. + Before you make commits, please install the git hooks provided in the repository. -```shell -script/configure-git-hooks.sh +```bash +bash script/configure-git-hooks.sh ``` This will ensure that your commits are formatted correctly and that the unittests pass before you push your changes. Be aware that the tests take a long time to run. If you want to skip the tests or formatting, you can use the `--no-verify` flag on `git commit` or `git push`. diff --git a/STATUS.md b/STATUS.md index 8af2a4e75..3b801d3aa 100644 --- a/STATUS.md +++ b/STATUS.md @@ -7,3 +7,11 @@ [![Client CI](https://github.com/into-cps-association/dtaas/actions/workflows/client.yml/badge.svg)](https://github.com/into-cps-association/dtaas/actions/workflows/client.yml/badge.svg) [![Server CI](https://github.com/into-cps-association/dtaas/actions/workflows/server.yml/badge.svg)](https://github.com/into-cps-association/dtaas/actions/workflows/server.yml/badge.svg) + +[![Contributors](https://img.shields.io/github/contributors/INTO-CPS-Association/DTaaS)](https://github.com/INTO-CPS-Association/DTaaS/graphs/contributors) + +[![Latest Commit](https://img.shields.io/github/last-commit/INTO-CPS-Association/DTaaS)](https://github.com/INTO-CPS-Association/DTaaS/commits/feature/distributed-demo) + +[![INTO-CPS Association](https://img.shields.io/badge/INTO_CPS_Association-white)](https://into-cps.org/) + + diff --git a/client/config/prod.js b/client/config/prod.js index 3c73cc2bf..86eabbaec 100644 --- a/client/config/prod.js +++ b/client/config/prod.js @@ -1,5 +1,5 @@ window.env = { - REACT_APP_ENVIRONMENT: 'dev', + REACT_APP_ENVIRONMENT: 'prod', REACT_APP_URL: 'https://foo.com/', REACT_APP_URL_BASENAME: 'dtaas', REACT_APP_URL_DTLINK: '/lab', @@ -15,4 +15,4 @@ window.env = { REACT_APP_REDIRECT_URI: 'https://foo.com/Library', REACT_APP_LOGOUT_REDIRECT_URI: 'https://foo.com/', REACT_APP_GITLAB_SCOPES: 'openid profile read_user read_repository api', -}; \ No newline at end of file +}; diff --git a/deploy/README.md b/deploy/README.md new file mode 100644 index 000000000..bfe84d6da --- /dev/null +++ b/deploy/README.md @@ -0,0 +1,93 @@ +# DTaaS on Linux Operating System + +This directory contains code for running DTaaS application on a Ubuntu Server 20.04 Operating System. The setup requires a machine which can spare 16GB RAM, 8 vCPUs and 50GB Hard Disk space. + +A dummy **foo.com** URL has been used for illustration. Please change this to your unique website URL. It is assumed that you are going to serve the application in only HTTPS mode. + +Please follow these steps to make this work in your local environment. Download the codebase as zip file into your computer and unzip the same into a directory named **DTaaS**. The rest of the instructions assume that your working directory is **DTaaS**. + +## Configuration + +You need to configure the gateway, library microservice and react client website. + +The first step is to decide on the number of users and their usenames. The traefik gateway configuration has a template for two users. You can modify the usernames in the template to the usernames chosen by you. + +### The traefik gateway server + +You can run the Run the Traefik gateway server in both and HTTPS and HTTPS mode to experience the DTaaS application. The installation guide assumes that you can run the application in HTTPS mode. + +The Traefik gateway configuration is at [fileConfig](../config/gateway/fileConfig.yml). Change `localhost` to `foo.com` and user1/user2 to the usernames chosen by you. + +**NOTE**: Do not use `http://` or `https://` in [fileConfig](../config/gateway/fileConfig.yml). + +#### Authentication + +The dummy username is `foo` and the password is `bar`. +Please change this before starting the gateway. + +```bash +rm deploy/config/gateway/auth +touch deploy/config/gateway/auth +htpasswd deploy/config/gateway/auth +password: +``` + +The user credentials added in [auth](../config/gateway/auth) should match the usernames in [fileConfig](../config/gateway/fileConfig.yml). + +## Configure lib microservice +The library microservice requires configuration. A template of this configuration file is given in _config/lib_ file. Please modify this file as per your needs. + +The first step in this configuration is to prepare the a filesystem for users. An example file system in `files/` directory. You can rename the top-level user1/user2 to the usernames chosen by you. + +Add an environment file named .env in lib for the library microservice. An example `.env` file is given below. The simplest possibility is to use `local` mode with the following example. The filepath is the absolute filepath to `files/` directory. You can copy this configuration into _config/lib_ file to get started. +```env +PORT='4001' +MODE='local' +LOCAL_PATH ='filepath' +LOG_LEVEL='debug' +APOLLO_PATH='/lib' +GRAPHQL_PLAYGROUND='true' +``` + +## Configure react website + +Change the React website configuration in _deploy/config/client/env.js_. + +```js +window.env = { + REACT_APP_ENVIRONMENT: 'prod', + REACT_APP_URL: 'https://foo.com/', + REACT_APP_URL_BASENAME: 'dtaas', + REACT_APP_URL_DTLINK: '/lab', + REACT_APP_URL_LIBLINK: '', + REACT_APP_WORKBENCHLINK_TERMINAL: '/terminals/main', + REACT_APP_WORKBENCHLINK_VNCDESKTOP: '/tools/vnc/?password=vncpassword', + REACT_APP_WORKBENCHLINK_VSCODE: '/tools/vscode/', + REACT_APP_WORKBENCHLINK_JUPYTERLAB: '/lab', + REACT_APP_WORKBENCHLINK_JUPYTERNOTEBOOK: '', + + REACT_APP_CLIENT_ID: '934b98f03f1b6f743832b2840bf7cccaed93c3bfe579093dd0942a433691ccc0', + REACT_APP_AUTH_AUTHORITY: 'https://gitlab.foo.com/', + REACT_APP_REDIRECT_URI: 'https://foo.com/Library', + REACT_APP_LOGOUT_REDIRECT_URI: 'https://foo.com/', + REACT_APP_GITLAB_SCOPES: 'openid profile read_user read_repository api', +}; +``` + +## Update the installation script + +Open `deploy/install.sh` and update user1/user2 to usernames chosen by you. + +## Perform the Installation + +Go to the DTaaS directory and execute + +```sh +source deploy/install.sh +``` + +You can run this script multiple times until the installation is successful. + +## Access the application + +Now you should be able to access the DTaaS application at: _https://foo.com_ diff --git a/deploy/config/client/env.js b/deploy/config/client/env.js new file mode 100644 index 000000000..dc97190eb --- /dev/null +++ b/deploy/config/client/env.js @@ -0,0 +1,18 @@ +window.env = { + REACT_APP_ENVIRONMENT: 'dev', + REACT_APP_URL: 'https://foo.com/', + REACT_APP_URL_BASENAME: 'dtaas', + REACT_APP_URL_DTLINK: '/lab', + REACT_APP_URL_LIBLINK: '', + REACT_APP_WORKBENCHLINK_TERMINAL: '/terminals/main', + REACT_APP_WORKBENCHLINK_VNCDESKTOP: '/tools/vnc/?password=vncpassword', + REACT_APP_WORKBENCHLINK_VSCODE: '/tools/vscode/', + REACT_APP_WORKBENCHLINK_JUPYTERLAB: '/lab', + REACT_APP_WORKBENCHLINK_JUPYTERNOTEBOOK: '', + + REACT_APP_CLIENT_ID: '934b98f03f1b6f743832b2840bf7cccaed93c3bfe579093dd0942a433691ccc0', + REACT_APP_AUTH_AUTHORITY: 'https://gitlab.foo.com/', + REACT_APP_REDIRECT_URI: 'https://foo.com/Library', + REACT_APP_LOGOUT_REDIRECT_URI: 'https://foo.com/', + REACT_APP_GITLAB_SCOPES: 'openid profile read_user read_repository api', +}; diff --git a/deploy/config/gateway/auth b/deploy/config/gateway/auth new file mode 100644 index 000000000..2e468b945 --- /dev/null +++ b/deploy/config/gateway/auth @@ -0,0 +1 @@ +foo:$apr1$fGdMVLcl$dMzJW.JF8Rn6Vzrf5uxaU/ diff --git a/deploy/vagrant/two-machine/fileConfig.yml b/deploy/config/gateway/fileConfig.yml similarity index 100% rename from deploy/vagrant/two-machine/fileConfig.yml rename to deploy/config/gateway/fileConfig.yml diff --git a/deploy/config/lib b/deploy/config/lib new file mode 100644 index 000000000..1acc5496d --- /dev/null +++ b/deploy/config/lib @@ -0,0 +1,9 @@ +PORT='4001' +MODE='local' or 'gitlab' +LOCAL_PATH ='filepath' +GITLAB_GROUP ='dtaas' +GITLAB_URL='https://gitlab.foo.com/api/graphql' +TOKEN='123-sample-token' +LOG_LEVEL='debug' +APOLLO_PATH='/lib' or '' +GRAPHQL_PLAYGROUND='false' or 'true' \ No newline at end of file diff --git a/deploy/create-cronjob.sh b/deploy/create-cronjob.sh new file mode 100755 index 000000000..ac396129e --- /dev/null +++ b/deploy/create-cronjob.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# Add a cronjob for the DTaaS services to the current users crontab +# the actual crontab script is cron.sh +# all existing crontab jobs are retained + +if [ -n "$1" ]; then + PROJECT_PATH="$1" +else + PROJECT_PATH="$(pwd)" +fi + +export PROJECT_PATH + +printf "Existing crontab for the user\n" +crontab -l +crontab -l | sort -u > temp.cron +echo "* * * * * ${PROJECT_PATH}/deploy/cron.sh ${PROJECT_PATH}" >> temp.cron +sort -u temp.cron > jobs.cron +crontab jobs.cron +rm jobs.cron temp.cron +printf "Updated crontab for the user\n" +crontab -l \ No newline at end of file diff --git a/deploy/cron.sh b/deploy/cron.sh new file mode 100644 index 000000000..e48fd2858 --- /dev/null +++ b/deploy/cron.sh @@ -0,0 +1,25 @@ +#!/bin/bash +# Run the client website and lib microservice in background mode + +PROJECT_PATH="$1" + +nc -z localhost 4000 +PORT_STATUS=$? +if (( PORT_STATUS == 1 )) +then + echo "starting react website" + cd "${PROJECT_PATH}/client" || exit + nohup serve -s build -l 4000 & disown +fi + + + +nc -z localhost 4001 +PORT_STATUS=$? +if (( PORT_STATUS == 1 )) +then + cd "${PROJECT_PATH}/servers/lib" || exit + nohup yarn start & disown +fi + +docker start traefik-gateway \ No newline at end of file diff --git a/deploy/install.sh b/deploy/install.sh new file mode 100755 index 000000000..ab15df110 --- /dev/null +++ b/deploy/install.sh @@ -0,0 +1,94 @@ +#!/bin/bash +set -eu + +printf "NOTE\n " +printf "....\n " +printf "This script installs DTaaS application for two users.\n " +printf "Remember to configure the application before proceeding further....\n " +printf "All the configuration instructions are in deploy/README.md file.\n " +printf ".........\n \n \n " + +printf "Press Ctl+C if you need to complete the configuration....\n " +printf "Waiting for 60 seconds....\n " +sleep 60 + +printf "\n \n Install the system dependencies...\n " +printf "....\n " +bash script/env.sh || exit + +printf "\n \n Download the required docker images...\n " +printf ".........\n " +source script/docker.sh || exit + + +printf "\n \n Continue with the DTaaS installation...\n " +printf ".........\n " +TOP_DIR=$(pwd) + +printf "\n \n Build, configure and run the react website\n " +printf ".....\n " +cd "${TOP_DIR}/client" || exit +yarn install +yarn build + +#one of the environments; specify only one; "dev" used the REACT_APP_ENV is not set +yarn configapp prod +cp "${TOP_DIR}/deploy/config/client/env.js" build/env.js +nohup serve -s build -l 4000 & disown + +#------------- +printf "\n\nStart the lib microservice\n " +printf "...........\n " +cd "${TOP_DIR}/servers/lib" || exit +yarn install +yarn build +cp "${TOP_DIR}/deploy/config/lib" .env +nohup yarn start & disown + +#------------- +printf "\n \n Start the user workspaces\n " +printf "...........\n " +docker run -d \ + -p 8090:8080 \ + --name "ml-workspace-user1" \ + -v "${TOP_DIR}/files/user1:/workspace" \ + -v "${TOP_DIR}/files/common:/workspace/common:ro" \ + --env AUTHENTICATE_VIA_JUPYTER="" \ + --env WORKSPACE_BASE_URL="user1" \ + --shm-size 512m \ + --restart always \ + mltooling/ml-workspace:0.13.2 || true + +docker run -d \ + -p 8091:8080 \ + --name "ml-workspace-user2" \ + -v "${TOP_DIR}/files/user2:/workspace" \ + -v "${TOP_DIR}/files/common:/workspace/common:ro" \ + --env AUTHENTICATE_VIA_JUPYTER="" \ + --env WORKSPACE_BASE_URL="user2" \ + --shm-size 512m \ + --restart always \ + mltooling/ml-workspace:0.13.2 || true + +#------------- +printf "\n \n Start the traefik gateway server\n " +printf "...........\n " +cd "${TOP_DIR}/servers/config/gateway" || exit +cp "${TOP_DIR}/deploy/config/gateway/auth" auth +cp "${TOP_DIR}/deploy/config/gateway/fileConfig.yml" "dynamic/fileConfig.yml" + +sudo docker run -d \ + --name "traefik-gateway" \ + --network=host -v "$PWD/traefik.yml:/etc/traefik/traefik.yml" \ + -v "$PWD/auth:/etc/traefik/auth" \ + -v "$PWD/dynamic:/etc/traefik/dynamic" \ + -v /var/run/docker.sock:/var/run/docker.sock \ + traefik:v2.5 || true + +#---------- +printf "\n \n Create crontabs to run the application in daemon mode.\n " +printf "...........\n " +cd "$TOP_DIR" || exit +bash deploy/create-cronjob.sh + +printf "\n \n The installation is complete.\n \n \n " diff --git a/deploy/single-script-install.sh b/deploy/single-script-install.sh new file mode 100755 index 000000000..07743c1fb --- /dev/null +++ b/deploy/single-script-install.sh @@ -0,0 +1,196 @@ +#!/bin/bash +set -eu + +printf "Install script for DTaaS software platform.\n" +printf "You can run the script multiple times until the installation succeeds.\n " +printf ".........\n \n \n " + +printf "Install the required system software dependencies...\n " +printf ".........\n \n \n " +sudo apt-get update -y +sudo apt-get upgrade -y + + +# Install docker for containers and microservices +# https://docs.docker.com/engine/install/ubuntu/ +sudo apt-get install -y \ + ca-certificates \ + curl \ + gnupg \ + lsb-release \ + zsh \ + apache2-utils \ + net-tools + +sudo mkdir -p /etc/apt/keyrings +if [ ! -f /etc/apt/keyrings/docker.gpg ] +then + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg + sudo printf \ + "deb [arch=%s signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ + %s stable" "$(dpkg --print-architecture)" "$(lsb_release -cs)" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null +fi + +sudo apt-get update -y +sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin +sudo groupadd docker || true +sudo usermod -aG docker "$USER" || true + +printf "\n\n\nMake docker available to your user account....\n " +printf ".......\n " +printf "Your user account is member of:\n " +groups +printf "groups.\n " + +printf "If your user account is a member of docker group, let the installation script continue.\n " +printf "Otherwise, exit this script and run the following command\n\n " +printf "sudo usermod -aG docker %s\n\n " "$USER" +printf "logout and login again. You can run this script again after login\n\n " +printf "Press Ctl+C if you need to complete the this task....\n " +printf "Waiting for 60 seconds....\n " +sleep 60 + +#newgrp docker +sudo service docker start +docker run hello-world + +sudo systemctl enable docker.service +sudo systemctl enable containerd.service + + +# Install nodejs environment +curl -fsSL https://deb.nodesource.com/setup_18.x | sudo bash - +sudo apt-get install -y nodejs + +if [ ! -f /usr/share/keyrings/yarnkey.gpg ] +then + curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | gpg --dearmor | sudo tee /usr/share/keyrings/yarnkey.gpg >/dev/null + printf "deb [signed-by=/usr/share/keyrings/yarnkey.gpg] https://dl.yarnpkg.com/debian stable main" | sudo tee /etc/apt/sources.list.d/yarn.list +fi + +sudo apt-get update -y +sudo apt-get install -y yarn +sudo npm install -g serve + +printf "\n\n End of installing dependencies...\n\n\n " +#---- + +# get the required docker images +printf "Download the required docker images...\n " +printf ".........\n\n\n " +docker pull traefik:v2.5 +docker pull influxdb:2.4 +docker pull mltooling/ml-workspace:0.13.2 +docker pull grafana/grafana +docker pull telegraf +docker pull gitlab/gitlab-ce:15.10.0-ce.0 +printf "\n\n docker images successfully downloaded...\n \n \n " +#---- + +printf "NOTE\n " +printf "....\n " +printf "This script installs DTaaS with default settings.\n " +printf "The setup is good for testing but not for secure installation.\n " + + +printf "\n\nClone the DTaaS codebase\n " +printf "...........\n " +if [ -d DTaaS ] +then + cd DTaaS || exit +else + git clone https://github.com/INTO-CPS-Association/DTaaS.git DTaaS + cd DTaaS || exit + git fetch --all + git checkout feature/distributed-demo +fi + +TOP_DIR=$(pwd) + +#------------- +printf "\n\n Build, configure and run the react website\n " +printf ".....\n " +cd "${TOP_DIR}/client" || exit +yarn install +yarn build + +yarn configapp dev +cp "${TOP_DIR}/deploy/config/client/env.js" build/env.js +nohup serve -s build -l 4000 & disown + +#------------- +printf "\n\n Build, configure and run the lib microservice\n " +printf "...........\n " +cd "${TOP_DIR}/servers/lib" || exit +yarn install +yarn build + +{ + printf "PORT='4001'\n " + printf "MODE='local'\n " + printf "LOCAL_PATH ='%s'\n " "$TOP_DIR" + printf "GITLAB_GROUP ='dtaas'\n " + printf "GITLAB_URL='https://gitlab.com/api/graphql'\n " + printf "TOKEN='123-sample-token'\n " + printf "LOG_LEVEL='debug'\n " + printf "APOLLO_PATH='lib'\n " + printf "GRAPHQL_PLAYGROUND='true'\n " +} > .env +nohup yarn start & disown + + +#------------- +printf "\n\n Start the user workspaces\n " +printf "...........\n " +docker run -d \ + -p 8090:8080 \ + --name "ml-workspace-user1" \ + -v "${TOP_DIR}/files/user1:/workspace" \ + -v "${TOP_DIR}/files/common:/workspace/common:ro" \ + --env AUTHENTICATE_VIA_JUPYTER="" \ + --env WORKSPACE_BASE_URL="user1" \ + --shm-size 512m \ + --restart always \ + mltooling/ml-workspace:0.13.2 || true + +docker run -d \ + -p 8091:8080 \ + --name "ml-workspace-user2" \ + -v "${TOP_DIR}/files/user2:/workspace" \ + -v "${TOP_DIR}/files/common:/workspace/common:ro" \ + --env AUTHENTICATE_VIA_JUPYTER="" \ + --env WORKSPACE_BASE_URL="user2" \ + --shm-size 512m \ + --restart always \ + mltooling/ml-workspace:0.13.2 || true + +#------------- +printf "\n\n Start the traefik gateway server\n " +printf "...........\n " +cd "${TOP_DIR}/servers/config/gateway" || exit +cp "${TOP_DIR}/deploy/config/gateway/auth" auth +cp "${TOP_DIR}/deploy/config/gateway/fileConfig.yml" "dynamic/fileConfig.yml" + +docker run -d \ + --name "traefik-gateway" \ + --network=host -v "$PWD/traefik.yml:/etc/traefik/traefik.yml" \ + -v "$PWD/auth:/etc/traefik/auth" \ + -v "$PWD/dynamic:/etc/traefik/dynamic" \ + -v /var/run/docker.sock:/var/run/docker.sock \ + traefik:v2.5 || true + + +#---------- +printf "\n\n Create crontabs to run the application in daemon mode.\n " +printf "...........\n " +cd "$TOP_DIR" || exit +bash deploy/create-cronjob.sh + +printf "\n\n The installation is complete.\n\n\n " + + +printf "Continue with the application configuration.\n " +printf ".........\n\n\n " +printf "Remember to change foo.com to your local hostname in the following files.\n " +printf "1. %s/client/build/env.js\n " "$TOP_DIR" +printf "2. %s/servers/config/gateway/dynamic/fileConfig.yml\n " "$TOP_DIR" diff --git a/deploy/vagrant/make_boxes/dtaas/README.md b/deploy/vagrant/make_boxes/dtaas/README.md index 631fc16cf..c9826405d 100644 --- a/deploy/vagrant/make_boxes/dtaas/README.md +++ b/deploy/vagrant/make_boxes/dtaas/README.md @@ -58,4 +58,4 @@ vagrant box add --name dtaas ./dtaas.vagrant ## TODO 1. Write a script for automating the above steps -1. Generate the ssh keys from ssl/certificates.bash \ No newline at end of file +1. Generate the ssh keys from ssl/certificates.bash diff --git a/deploy/vagrant/make_boxes/dtaas/provision.sh b/deploy/vagrant/make_boxes/dtaas/provision.sh index 2653e2b8b..9243314ef 100755 --- a/deploy/vagrant/make_boxes/dtaas/provision.sh +++ b/deploy/vagrant/make_boxes/dtaas/provision.sh @@ -30,13 +30,9 @@ docker run hello-world systemctl enable docker.service systemctl enable containerd.service -# get the required docker images -docker pull traefik:v2.5 -docker pull influxdb:2.4 -docker pull mltooling/ml-workspace:0.13.2 -docker pull grafana/grafana -docker pull telegraf -docker pull gitlab/gitlab-ce:15.10.0-ce.0 +#install docker-compose from https://docs.docker.com/compose/install/other/ +sudo curl -SL "https://github.com/docker/compose/releases/download/v2.15.1/docker-compose-linux-x86_64" -o /usr/local/bin/docker-compose +sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose curl -fsSL https://deb.nodesource.com/setup_18.x | bash - apt-get install -y nodejs @@ -46,8 +42,16 @@ apt-get update -y apt-get install -y yarn npm install -g serve -apt install -y python3-pip +#------------- +printf "\n\n Install jupyter toolchain" +sudo apt-get install -y python3-pip sudo -H pip install jupyterlab +sudo -H pip install mkdocs +sudo -H pip3 install mkdocs-material +sudo -H pip3 install python-markdown-math +sudo -H pip3 install mkdocs-open-in-new-tab +sudo -H pip3 install mkdocs-with-pdf +sudo -H pip3 install qrcode # Install minimal Kubernetes cluster snap install microk8s --classic @@ -59,4 +63,11 @@ cat /vagrant/vagrant.pub >> /home/vagrant/.ssh/authorized_keys mkdir -p /root/.ssh cat /vagrant/vagrant.pub >> /root/.ssh/authorized_keys -printf "vagrant ALL=(ALL) NOPASSWD: ALL\n" >> /etc/sudoers +# get the required docker images +docker pull traefik:v2.5 +docker pull influxdb:2.4 +docker pull mltooling/ml-workspace:0.13.2 +docker pull grafana/grafana +docker pull telegraf +docker pull gitlab/gitlab-ce:15.10.0-ce.0 + diff --git a/deploy/vagrant/route.sh b/deploy/vagrant/route.sh new file mode 100755 index 000000000..5030fa9be --- /dev/null +++ b/deploy/vagrant/route.sh @@ -0,0 +1,12 @@ +#!/bin/bash +# remove the default route installed by vagrant + +printf "Existing crontab for root\n" +crontab -l +crontab -l | sort -u > temp.cron +echo "* * * * * ip route del default via 10.0.2.2 dev enp0s3" >> temp.cron +sort -u temp.cron > jobs.cron +crontab jobs.cron +rm jobs.cron temp.cron +printf "Updated crontab for root\n" +crontab -l \ No newline at end of file diff --git a/deploy/vagrant/single-machine/README.md b/deploy/vagrant/single-machine/README.md index d7408ca0d..c9b8afc5d 100644 --- a/deploy/vagrant/single-machine/README.md +++ b/deploy/vagrant/single-machine/README.md @@ -6,12 +6,11 @@ A dummy **foo.com** URL has been used for illustration. Please change this to yo Please follow these steps to make this work in your local environment. -1. Create [**dtaas** Vagrant box](../make_boxes/dtaas/README.md). Copy _vagrant_ SSH private key here. This shall be useful for logging into the vagrant machine create for single-machine deployment. You would have created an SSH key pair - _vagrant_ and _vagrant.pub_. The _vagrant_ is the private SSH key and is needed for the next steps. +1. Create [**dtaas** Vagrant box](../make_boxes/dtaas/README.md). You would have created an SSH key pair - _vagrant_ and _vagrant.pub_ for the vagrant box. The _vagrant_ is the private SSH key and is needed for the next steps. Copy _vagrant_ SSH private key into the current directory (`deploy/vagrant/single-machine`). This shall be useful for logging into the vagrant machine created for single-machine deployment. 1. Update the **Vagrantfile**. Fields to update are: 1. Hostname (`node.vm.hostname = "foo.com"`) 1. MAC address (`:mac => "xxxxxxxx"`). This change is required if you have a DHCP server assigning domain names based on MAC address. Otherwise, you can leave this field unchanged. 1. Other adjustments are optional. - 1. Copy _vagrant_ SSH private key into the current directory (`deploy/vagrant/single-machine`). 1. Execute the following commands from terminal ```bash @@ -19,24 +18,11 @@ vagrant up vagrant ssh ``` -The Traefik gateway configuration file will be at `/home/vagrant/DTaaS/servers/config/gateway/dynamic/fileConfig.yml`. Update it as per instructions in this [README](../../../servers/config/gateway/README.md). - -Change the React website configuration in _client/build/env.js_. - -```js -window.env = { - REACT_APP_ENVIRONMENT: 'development', - REACT_APP_URL_LIB: 'http://foo.com/user1/shared/filebrowser/files/workspace/?token=admin', - REACT_APP_URL_DT: 'http://foo.com/user1/lab', - REACT_APP_URL_WORKBENCH: 'http://foo.com/user1', -}; -``` - -Serve the react website. From inside the vagrant machine, +Set a cronjob inside the vagrant virtual machine to remote the conflicting default route. ```bash -cd ~/DTaaS/client -nohup serve -s build -l 4000 & disown +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh +sudo bash route.sh ``` -Now you should be able to access the DTaaS application at: _http://foo.com_ +Follow the [instructions](../../README.md) of regular server installation setup to complete the installation. diff --git a/deploy/vagrant/single-machine/Vagrantfile b/deploy/vagrant/single-machine/Vagrantfile index 7c1338087..16ae2352b 100644 --- a/deploy/vagrant/single-machine/Vagrantfile +++ b/deploy/vagrant/single-machine/Vagrantfile @@ -18,12 +18,7 @@ Vagrant.configure(2) do |config| vb.cpus = 8 end - # remove the incorrect default gateway - config.vm.provision "shell", - inline: "ip route del default via 10.0.2.2 dev enp0s3" - node.ssh.private_key_path = "vagrant" - node.vm.provision "shell", privileged: false, path: "start.sh" end end diff --git a/deploy/vagrant/single-machine/start.sh b/deploy/vagrant/single-machine/start.sh deleted file mode 100755 index 6ba924c7d..000000000 --- a/deploy/vagrant/single-machine/start.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/bin/bash - -git clone https://github.com/INTO-CPS-Association/DTaaS.git DTaaS -cd DTaaS || exit -TOP_DIR=`pwd` -git fetch --all -git checkout feature/distributed-demo - -#------------- -printf "\n\n start the react website" -cd "${TOP_DIR}/client" || exit -yarn install -yarn build - -#one of the environments; specify only one; "dev" used the REACT_APP_ENV is not set -yarn configapp dev -nohup serve -s build -l 4000 & disown - -#------------- -printf "\n\n start the jupyter notebook server" -docker run -d \ - -p 8090:8080 \ - --name "ml-workspace-user1" \ - -v "${TOP_DIR}/files/user1:/workspace" \ - -v "${TOP_DIR}/files/common:/workspace/common:ro" \ - --env AUTHENTICATE_VIA_JUPYTER="" \ - --env WORKSPACE_BASE_URL="user1" \ - --shm-size 512m \ - --restart always \ - mltooling/ml-workspace:0.13.2 - -#------------- -printf "\n\n start the traefik gateway server" -printf ".........................." -cd "${TOP_DIR}/servers/config/gateway" || exit -sudo docker run -d \ - --name "traefik-gateway" \ - --network=host -v "$PWD/traefik.yml:/etc/traefik/traefik.yml" \ - -v "$PWD/auth:/etc/traefik/auth" \ - -v "$PWD/dynamic:/etc/traefik/dynamic" \ - -v /var/run/docker.sock:/var/run/docker.sock \ - traefik:v2.5 diff --git a/deploy/vagrant/two-machine/README.md b/deploy/vagrant/two-machine/README.md index 6e32c696c..b5af66f63 100644 --- a/deploy/vagrant/two-machine/README.md +++ b/deploy/vagrant/two-machine/README.md @@ -20,13 +20,13 @@ Copy _vagrant_ SSH private key into the current directory (`deploy/vagrant/two-m ## Configure Server Settings -**NOTE**: A dummy **foo.com** URL has been used for illustration. Please change this to your unique website URL. +**NOTE**: A dummy **foo.com** and **services.foo.com** URLs has been used for illustration. Please change these to your unique website URLs. The first step is to define the network identity of the two VMs. For that, you need _server name_, _hostname_ and _MAC address_. The hostname is the network URL at which the server can be accessed on the WWW. Please follow these steps to make this work in your local environment. Update the **boxes.json**. There are entries one for each server. The fields to update are: - 1. `name` - name of server1 (`"name" = "workspaces"`) + 1. `name` - name of server1 (`"name" = "dtaas"`) 1. `hostname` - hostname of server1 (`"name" = "foo.com"`) 1. MAC address (`:mac => "xxxxxxxx"`). This change is required if you have a DHCP server assigning domain names based on MAC address. Otherwise, you can leave this field unchanged. 1. `name` - name of server2 (`"name" = "services"`) @@ -34,97 +34,45 @@ Update the **boxes.json**. There are entries one for each server. The fields to 1. MAC address (`:mac => "xxxxxxxx"`). This change is required if you have a DHCP server assigning domain names based on MAC address. Otherwise, you can leave this field unchanged. 1. Other adjustments are optional. -## Launch Server1 - -Execute the following commands from terminal - -```bash -vagrant up --provision server1 -vagrant ssh -``` - -The default Traefik gateway configuration file is available at `/home/vagrant/DTaaS/servers/config/gateway/dynamic/fileConfig.yml`. This has been configured for single user workspace.- The _fileConfig.yml_ in this directory is suitable for two machine setup. Overwrite the default config with this one. - -```bash -# From top-level of project directory -cp deploy/vagrant/two-machine/fileConfig.yml \ - servers/config/gateway/dynamic/fileConfig.yml -``` - -In the two machine setup, the following background services are offered through Traefik gateway. - -| service | internal url | external url | server hosting the service | -|:---|:---|:---|:---| -| react website | localhost:4000 | foo.com | server1 | -| user1 workspace | localhost:8090 | foo.com/user1 | server1 | -| user2 workspace | localhost:8091 | foo.com/user2 | server1 | -| grafana visualization service | localhost:3000 | foo.com/vis | server2 | -|| - -server2 also hosts InfluxDB and RabbitMQ services, but these are not serviced via Traefik gateway. - -Update _auth_ and _fileConfig.yml_ of Traefik gateway as per instructions in this [README](../../../servers/config/gateway/README.md). - -Change the React website configuration in _client/build/env.js_. - -```js -window.env = { - REACT_APP_ENVIRONMENT: 'development', - REACT_APP_URL_LIB: 'http://foo.com/user1/shared/filebrowser/files/workspace/?token=admin', - REACT_APP_URL_DT: 'http://foo.com/user1/lab', - REACT_APP_URL_WORKBENCH: 'http://foo.com/user1', -}; -``` - -Serve the react website. From inside the vagrant machine, - -```bash -cd ~/DTaaS/client -nohup serve -s build -l 4000 & disown -``` - -Now you should be able to access the DTaaS application at: _http://foo.com_ - -Each user gets a dedicated workspaces. Two users have been provisioned in this default setup. You can update the configuration to have more users. All the users have the same password, please keep this in mind while allowing more users. - -The following URLs must work now: - -* http://foo.com (website; by default this is configured for a single user) -* http://foo.com/user1 (user1 workspace) -* http://foo.com/user2 (user2 workspace) - -## Launch Server2 +## Launch platform default services RabbitMQ, Grafana and InfluxDB services are provisioned on this server. -InfluxDB webUI will be available at: server2.foo.com. The Grafana shall be accessible via server1 at _http://foo.com/vis_. +InfluxDB webUI will be available at: _services.foo.com_. The RabbitMQ service and its management interface shall be available at 5672 and 15672 TCP ports respectively. The Grafana service shall be available at TCP port 3000. + +The firewall and network access settings of corporate / cloud network need to be configured to allow external access to the services. Otherwise the users of DTaaS will not be able to utilize these services from their user workspaces. -The InfluxDB, Grafana and RabbitMQ services shall be run on this server. First, execute the following commands from terminal to start the machine. +Execute the following commands from terminal to start the machine. ```bash -vagrant up --provision server2 -vagrant ssh +vagrant up --provision services +vagrant ssh services +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/two-machine/services.sh +bash services.sh +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh +sudo bash route.sh ``` After the server is up and running, you can see the following services active within server2. -| service | internal url | external url | server hosting the service | -|:---|:---|:---|:---| -| Influx visualization service | localhost:80 | server2.foo.com | server2 | -| grafana visualization service | localhost:3000 | foo.com/vis | server2 | -| RabbitMQ communication service | localhost:5672 | not available | server2 | +| service | external url | +|:---|:---| +| Influx visualization service | services.foo.com | +| Grafana visualization service | services.foo.com:3000 | +| RabbitMQ communication service | services.foo.com:5672 | +| RabbitMQ management service | services.foo.com:15672 | || -All these services are available to users and machines with SSH access to server2. +## Launch DTaaS application -## Linking The Two Servers - -The services running on server2 must be made available to the user workspaces running on server1. Hence SSH commands need to be executed on server1 to perform remote port fowarding from server2 to server1. Log into server1 and perform: +Execute the following commands from terminal ```bash -cd ~/DTaaS/deploy/vagrant/two-machine -./link.sh +vagrant up --provision dtaas +vagrant ssh dtaas +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh +sudo bash route.sh ``` -The following URLs must work now: -* http://foo.com/vis (Grafana visualization service) -* http://server2.foo.com (Influx service) +If you only want to test the application and are not setting up a production instance, you can install using [single script install](../../single-script-install.sh). + +If you are not in a hurry and would rather have a production instance, follow the [instructions](../../README.md) of regular server installation setup to complete the installation. diff --git a/deploy/vagrant/two-machine/Vagrantfile b/deploy/vagrant/two-machine/Vagrantfile index 2902646e7..24eaeb3e1 100644 --- a/deploy/vagrant/two-machine/Vagrantfile +++ b/deploy/vagrant/two-machine/Vagrantfile @@ -21,9 +21,6 @@ Vagrant.configure(2) do |config| node.ssh.private_key_path = "vagrant" - # remove the incorrect default gateway - node.vm.provision "shell", path: "route.sh" - node.vm.provision "shell", privileged: false, path: opts["script"] end end diff --git a/deploy/vagrant/two-machine/boxes.json b/deploy/vagrant/two-machine/boxes.json index 23bbcfd9e..e47a5dc42 100644 --- a/deploy/vagrant/two-machine/boxes.json +++ b/deploy/vagrant/two-machine/boxes.json @@ -1,17 +1,15 @@ [ { - "name" : "server1", + "name" : "dtaas", "hostname" : "foo.com", - "script": "workspaces.sh", "mac" : "02163e0d3c28", "ssh_port" : 11022, "http_port" : 11080, "https_port" : 11443 }, { - "name" : "server2", - "hostname" : "server2.foo.com", - "script": "services.sh", + "name" : "services", + "hostname" : "services.foo.com", "mac" : "02163e8caced", "ssh_port" : 12022, "http_port" : 12080, diff --git a/deploy/vagrant/two-machine/link.sh b/deploy/vagrant/two-machine/link.sh deleted file mode 100755 index 63922c6d2..000000000 --- a/deploy/vagrant/two-machine/link.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash -# access the services on server2 from server1 -# to be executed on server1 -# RabbitMQ -ssh -i /vagrant/vagrant -fNT -L 15672:localhost:15672 vagrant@server2.foo.com -ssh -i /vagrant/vagrant -fNT -L 5672:localhost:5672 vagrant@server2.foo.com - -#InfluxDB -ssh -i /vagrant/vagrant -fNT -L 40000:localhost:80 vagrant@server2.foo.com -#Grafana -ssh -i /vagrant/vagrant -fNT -L 3000:localhost:3000 vagrant@server2.foo.com diff --git a/deploy/vagrant/two-machine/route.sh b/deploy/vagrant/two-machine/route.sh deleted file mode 100755 index 0bdbd7c2f..000000000 --- a/deploy/vagrant/two-machine/route.sh +++ /dev/null @@ -1,2 +0,0 @@ -#!/bin/bash -ip route del default via 10.0.2.2 dev enp0s3 diff --git a/deploy/vagrant/two-machine/services.sh b/deploy/vagrant/two-machine/services.sh index c0e54ece9..c68546e9c 100755 --- a/deploy/vagrant/two-machine/services.sh +++ b/deploy/vagrant/two-machine/services.sh @@ -15,12 +15,14 @@ docker exec rabbitmq-server rabbitmqctl set_permissions -p "/" incubator ".*" ". #start Grafana server docker run -d \ -p 3000:3000 \ - --name=grafana-test \ + --name=grafana \ -e "GF_SERVER_SERVE_FROM_SUB_PATH=true" \ - -e "GF_SERVER_DOMAIN=foo.com" \ - -e "GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/vis" \ + -e "GF_SERVER_DOMAIN=localhost" \ + -e "GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s" \ -e "GF_AUTH_BASIC_ENABLED=false" \ -e "GF_AUTH_PROXY_ENABLED=false" \ + -e "GF_SECURITY_ADMIN_PASSWORD=DTaaSGrafana" \ + -e "GF_SECURITY_ALLOW_EMBEDDING=true" \ -e "GF_SECURITY_ALLOW_EMBEDDING=true" \ -e "GF_AUTH_ANONYMOUS_ENABLED=true" \ -e "GF_AUTH_ANONYMOUS_ORG_NAME=Main" \ @@ -34,7 +36,7 @@ docker run -d \ -e "GF_PATHS_PLUGINS=/var/lib/grafana/plugins" \ -e "GF_PATHS_PROVISIONING=/etc/grafana/provisioning" \ -e "HOME=/home/grafana" \ - grafana/grafana + grafana/grafana printf "Complete the setup from GUI" #------------- diff --git a/deploy/vagrant/two-machine/workspaces.sh b/deploy/vagrant/two-machine/workspaces.sh deleted file mode 100755 index 3d437d3dc..000000000 --- a/deploy/vagrant/two-machine/workspaces.sh +++ /dev/null @@ -1,55 +0,0 @@ -#!/bin/bash - -git clone https://github.com/INTO-CPS-Association/DTaaS.git DTaaS -cd DTaaS || exit -TOP_DIR=$(pwd) -git fetch --all -git checkout feature/distributed-demo - -#------------- -printf "\n\n start the react website" -cd "${TOP_DIR}/client" || exit -yarn install -yarn build - -#one of the environments; specify only one; "dev" used the REACT_APP_ENV is not set -yarn configapp dev -nohup serve -s build -l 4000 & disown - -#------------- -printf "\n\n start the jupyter notebook server" -docker run -d \ - -p 8090:8080 \ - --name "ml-workspace-user1" \ - -v "${TOP_DIR}/files/user1:/workspace" \ - -v "${TOP_DIR}/files/common:/workspace/common:ro" \ - --env AUTHENTICATE_VIA_JUPYTER="" \ - --env WORKSPACE_BASE_URL="user1" \ - --shm-size 512m \ - --restart always \ - mltooling/ml-workspace:0.13.2 - -#------------- -printf "\n\n start the jupyter notebook server" -docker run -d \ - -p 8091:8080 \ - --name "ml-workspace-user2" \ - -v "${TOP_DIR}/files/user2:/workspace" \ - -v "${TOP_DIR}/files/common:/workspace/common:ro" \ - --env AUTHENTICATE_VIA_JUPYTER="" \ - --env WORKSPACE_BASE_URL="user2" \ - --shm-size 512m \ - --restart always \ - mltooling/ml-workspace:0.13.2 - -#------------- -printf "\n\n start the traefik gateway server" -printf ".........................." -cd "${TOP_DIR}/servers/config/gateway" || exit -sudo docker run -d \ - --name "traefik-gateway" \ - --network=host -v "$PWD/traefik.yml:/etc/traefik/traefik.yml" \ - -v "$PWD/auth:/etc/traefik/auth" \ - -v "$PWD/dynamic:/etc/traefik/dynamic" \ - -v /var/run/docker.sock:/var/run/docker.sock \ - traefik:v2.5 diff --git a/docs/FAQ.md b/docs/FAQ.md index a3ea18785..061371278 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -13,6 +13,7 @@ DTaaS is software platform on which you can create and run digital twins. Please see the [features](user/features.md) page to get a sense of the things you can do in DaaS. + ??? Question "Are there any Key Performance / Capability Indicators for DTaaS?" | Key Performance Indicator | Value | @@ -28,6 +29,10 @@ | Availability | It is a matter of human resources. If you have human resources to maintain DTaaS round the clock, upwards 95% is easily possible. | +??? Question "Do you provide licensed software like Matlab?" + + The licensed software are not available on the software platform. But users have private workspaces which are based on Linux-based xfce Desktop environment. Users can install software in their workspaces. The licensed software installed by one user is not available to another user. + ## Digital Twin Models diff --git a/docs/admin/client/CLIENT.md b/docs/admin/client/CLIENT.md index 46ecacbf6..5afb4f5ec 100644 --- a/docs/admin/client/CLIENT.md +++ b/docs/admin/client/CLIENT.md @@ -2,11 +2,11 @@ To host DTaaS client website on your server, follow these steps: -- The `build` folder contains all the optimized static files that are ready for deployment. +- Download the **DTaaS-client.zip** from the [releases page](https://github.com/INTO-CPS-Association/DTaaS/releases). +- Inside the `DTaaS-client` directory, there is `site` directory. The `site` directory contains all the optimized static files that are ready for deployment. - Setup the oauth application on gitlab instance. See the instructions in [authentication page](auth.md) for completing this task. - -- Locate the file `build/env.js` and replace the example values to match your infrastructure. The constructed links will be "`REACT_APP_URL`/`REACT_APP_URL_BASENAME`/`{username}`/`{Endpoint}`". See the definitions below: +- Locate the file `site/env.js` and replace the example values to match your infrastructure. The constructed links will be "`REACT_APP_URL`/`REACT_APP_URL_BASENAME`/`{username}`/`{Endpoint}`". See the definitions below: ```js window.env = { @@ -31,7 +31,7 @@ To host DTaaS client website on your server, follow these steps: // Example values with no base URL. Trailing and ending slashes are optional. window.env = { - REACT_APP_ENVIRONMENT: 'dev', + REACT_APP_ENVIRONMENT: 'prod', REACT_APP_URL: 'https://foo.com/', REACT_APP_URL_BASENAME: '', REACT_APP_URL_DTLINK: '/lab', @@ -77,4 +77,48 @@ To host DTaaS client website on your server, follow these steps: - Once the files are on your server, you should be able to access your app by visiting your server's IP address or domain name in a web browser. -:fontawesome-solid-circle-info: The website depends on Traefik and ML Workspace components to be available. Otherwise, you only get a skeleton non-functional website. \ No newline at end of file +:fontawesome-solid-circle-info: The website depends on **Traefik gateway** and **ML Workspace** components to be available. Otherwise, you only get a skeleton non-functional website. + +## Complementary Components + +The website requires background services for providing actual functionality. The minimum background service required is atleast one [ML Workspace](https://github.com/ml-tooling/ml-workspace) serving the following routes. + +```js +https:foo.com//lab +https:foo.com//terminals/main +https:foo.com//tools/vnc/?password=vncpassword +https:foo.com//tools/vscode/ +``` + +The `username` is the user workspace created using ML Workspace docker container. Please follow the instructions in [README](https://github.com/ml-tooling/ml-workspace/blob/main/README.md). You can create as many user workspaces as you want. If you have two users - alice and bob - on your system, then the following the commands in will instantiate the required user workspaces. + +```bash +mkdir -p files/alice files/bob files/common + +printf "\n\n start the user workspaces" +docker run -d \ + -p 8090:8080 \ + --name "ml-workspace-alice" \ + -v "$(pwd)/files/alice:/workspace" \ + -v "$(pwd)/files/common:/workspace/common" \ + --env AUTHENTICATE_VIA_JUPYTER="" \ + --env WORKSPACE_BASE_URL="alice" \ + --shm-size 512m \ + --restart always \ + mltooling/ml-workspace:0.13.2 + + + +docker run -d \ + -p 8091:8080 \ + --name "ml-workspace-bob" \ + -v "$(pwd)/files/bob:/workspace" \ + -v "$(pwd)/files/common:/workspace/common" \ + --env AUTHENTICATE_VIA_JUPYTER="" \ + --env WORKSPACE_BASE_URL="bob" \ + --shm-size 512m \ + --restart always \ + mltooling/ml-workspace:0.13.2 +``` + +Given that multiple services are running at different routes, a reverse proxy is needed to map the background services to external routes. You can use Apache, NGINX, Traefik or any other software to work as reverse proxy. diff --git a/docs/admin/host.md b/docs/admin/host.md new file mode 100644 index 000000000..372ed90b3 --- /dev/null +++ b/docs/admin/host.md @@ -0,0 +1,99 @@ +# DTaaS on Linux Operating System + +These are installation instructions for running DTaaS application on a Ubuntu Server 20.04 Operating System. The setup requires a machine which can spare 16GB RAM, 8 vCPUs and 50GB Hard Disk space. + +A dummy **foo.com** URL has been used for illustration. Please change this to your unique website URL. It is assumed that you are going to serve the application in only HTTPS mode. + +Please follow these steps to make this work in your local environment. Download the **DTaaS.zip** from the [releases page](https://github.com/INTO-CPS-Association/DTaaS/releases). Unzip the same into a directory named **DTaaS**. The rest of the instructions assume that your working directory is **DTaaS**. + +!!! note + If you only want to test the application and are not setting up a production instance, you can follow the instructions of [trial installation](trial.md). + +## Configuration + +You need to configure the Traefik gateway, library microservice and react client website. + +The first step is to decide on the number of users and their usenames. The traefik gateway configuration has a template for two users. You can modify the usernames in the template to the usernames chosen by you. + +### The traefik gateway server + +You can run the Run the Traefik gateway server in both HTTP and HTTPS mode to experience the DTaaS application. The installation guide assumes that you can run the application in HTTPS mode. + +The Traefik gateway configuration is at _deploy/config/gateway/fileConfig.yml_. Change `foo.com` to your local hostname and user1/user2 to the usernames chosen by you. + +!!! tip + + Do not use `http://` or `https://` in _deploy/config/gateway/fileConfig.yml_. + +#### Authentication + +The dummy username is `foo` and the password is `bar`. +Please change this before starting the gateway. + +```bash +rm deploy/config/gateway/auth +touch deploy/config/gateway/auth +htpasswd deploy/config/gateway/auth +password: +``` + +The user credentials added in _deploy/config/gateway/auth_ should match the usernames in _deploy/config/gateway/fileConfig.yml_. + +## Configure lib microservice +The library microservice requires configuration. A template of this configuration file is given in _deploy/config/lib_ file. Please modify this file as per your needs. + +The first step in this configuration is to prepare the a filesystem for users. An example file system in `files/` directory. You can rename the top-level user1/user2 to the usernames chosen by you. + +An example `.env` file is given below. The simplest possibility is to use `local` mode with the following example. The filepath is the absolute filepath to `files/` directory. You can copy this configuration into _deploy/config/lib_ file to get started. + +```env +PORT='4001' +MODE='local' +LOCAL_PATH ='filepath' +LOG_LEVEL='debug' +APOLLO_PATH='/lib' +GRAPHQL_PLAYGROUND='true' +``` + +## Configure react website + +Change the React website configuration in _deploy/config/client/env.js_. + +```js +window.env = { + REACT_APP_ENVIRONMENT: 'dev', + REACT_APP_URL: 'https://foo.com/', + REACT_APP_URL_BASENAME: 'dtaas', + REACT_APP_URL_DTLINK: '/lab', + REACT_APP_URL_LIBLINK: '', + REACT_APP_WORKBENCHLINK_TERMINAL: '/terminals/main', + REACT_APP_WORKBENCHLINK_VNCDESKTOP: '/tools/vnc/?password=vncpassword', + REACT_APP_WORKBENCHLINK_VSCODE: '/tools/vscode/', + REACT_APP_WORKBENCHLINK_JUPYTERLAB: '/lab', + REACT_APP_WORKBENCHLINK_JUPYTERNOTEBOOK: '', + + REACT_APP_CLIENT_ID: '934b98f03f1b6f743832b2840bf7cccaed93c3bfe579093dd0942a433691ccc0', + REACT_APP_AUTH_AUTHORITY: 'https://gitlab.foo.com/', + REACT_APP_REDIRECT_URI: 'https://foo.com/Library', + REACT_APP_LOGOUT_REDIRECT_URI: 'https://foo.com/', + REACT_APP_GITLAB_SCOPES: 'openid profile read_user read_repository api', +}; +``` + +## Update the installation script + +Open `deploy/install.sh` and update user1/user2 to usernames chosen by you. + +## Perform the Installation + +Go to the DTaaS directory and execute + +```sh +source deploy/install.sh +``` + +You can run this script multiple times until the installation is successful. + +## Access the application + +Now you should be able to access the DTaaS application at: _https://foo.com_ diff --git a/docs/admin/servers/lib/LIB-MS.md b/docs/admin/servers/lib/LIB-MS.md index 1e475e8d2..151d5ea28 100644 --- a/docs/admin/servers/lib/LIB-MS.md +++ b/docs/admin/servers/lib/LIB-MS.md @@ -1,8 +1,14 @@ # Host Library Microservice +The **lib microservice** is a simplified file manager providing graphQL API. It has three features: + +* provide a listing of directory contents. +* transfer a file to user. +* Source files can either come from local file system or from a gitlab instance. + The library microservice is designed to manage and serve files, functions, and models to users, allowing them to access and interact with various resources. -This document provides instructions for running the library microservice. +This document provides instructions for running a stand alone library microservice. ## Setup the File System @@ -11,19 +17,25 @@ The users expect the following file system structure for their reusable assets. ![File System Layout](file-system-layout.png) There is a skeleton file structure in [DTaaS codebase](https://github.com/INTO-CPS-Association/DTaaS/tree/feature/distributed-demo/files). You can copy and create file system for your users. + +## Gitlab setup (optional) + +For this microserivce to be functional, a certain directory or gitlab project structure is expected. The microservice expects that the gitlab consisting of one group, DTaaS, and within that group, all of the projects be located, **user1**, **user2**, ... , as well as a **commons** project. Each project corresponds to files of one user. +A sample file structure can be seen in [gitlab dtaas group](https://gitlab.com/dtaas). You can visit the gitlab documentation on [groups](https://docs.gitlab.com/ee/user/group/) for help on the management of gitlab groups. + +You can clone the git repositories from the `dtaas` group to get a sample file system structure for the lib microservice. + ## Setup Microservice To set up the lib microservice, follow these steps: -```sh -git clone https://github.com/INTO-CPS-Association/DTaaS.git -cd DTaaS/server/lib -yarn install # Install the required dependencies -``` +Download the **lib-microservice.zip** from the [releases page](https://github.com/INTO-CPS-Association/DTaaS/releases). + +## Configuration setup -### Environment Variables +The microservices uses `.env` environment files to receive configuration. -To set up the environment variables for the lib microservice, create a new file named _.env_ in the `servers/lib` folder. Then, add the following variables and their respective values. Below you can see and how, with included examples: +To set up the environment variables for the lib microservice, create a new file named _.env_ in the `lib-ms` directory. Then, add the following variables and their respective values. Below you can see and how, with included examples: ```ini PORT='4001' @@ -32,10 +44,17 @@ LOCAL_PATH='/Users//DTaaS/files' GITLAB_GROUP='dtaas' GITLAB_URL='https://gitlab.com/api/graphql' TOKEN='123-sample-token' +LOG_LEVEL='debug' APOLLO_PATH='/lib' or '' GRAPHQL_PLAYGROUND='false' or 'true' ``` +The `LOCAL_PATH` variable is the absolute filepath to the location of the local directory which will be served to users by the Library microservice. + +The `GITLAB_URL`, `GITLAB_GROUP` and `TOKEN` are only relevant for `gitlab` mode. The `TOKEN` should be set to your GitLab Group access API token. For more information on how to create and use your access token, [gitlab page](https://docs.gitlab.com/ee/user/group/settings/group_access_tokens.html). + +Once you've generated a token, copy it and replace the value of `TOKEN` with your token for the gitlab group, can be found. + Replace the default values the appropriate values for your setup. **NOTE**: @@ -46,19 +65,22 @@ Replace the default values the appropriate values for your setup. ### Start Microservice ```bash -yarn install -yarn build -yarn start +yarn install # Install dependencies for the microservice +yarn build # build the application +yarn start # start the application +``` + +You can press `Ctl+C` to halt the application. If you wish to run the microservice in the background, use + +```bash +nohup yarn start & disown ``` The lib microservice is now running and ready to serve files, functions, and models. Users can access the library microservice at URL: `http://localhost:/lib`. +## Service Endpoint -### Modes of Operation - -The library microservice hides the mode of operation from the users. The files can come from two sources: +The URL endpoint for this microservice is located at: `localhost:PORT/lib` -1. A directory on the server machine hosting DTaaS (`mode=local`) -1. A gitlab group on a gitlab instance (`mode=gitlab`). The gitlab can either be a community [gitlab](https://gitlab.com) or your own private gitlab instance. \ No newline at end of file diff --git a/docs/admin/trial.md b/docs/admin/trial.md new file mode 100644 index 000000000..308103594 --- /dev/null +++ b/docs/admin/trial.md @@ -0,0 +1,13 @@ + +# Trial Installation + +A single step install script is helpful in performing a trial run of the software. This script installs DTaaS software with default credentials and users on a Ubuntu server OS. You can use it to check a test installation of DTaaS. + +```bash +wget https://github.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/single-script-install.sh +bash single-script-install.sh +``` + +!!! warning + + This test installation has default credentials and is thus highly insecure. diff --git a/docs/admin/vagrant/base-box.md b/docs/admin/vagrant/base-box.md new file mode 100644 index 000000000..68ccd319d --- /dev/null +++ b/docs/admin/vagrant/base-box.md @@ -0,0 +1,66 @@ +# Vagrant Box for DTaaS + +There are some good vagrant boxes on [vagrant website](https://app.vagrantup.com/boxes/search). But these boxes require too many installations that take a long time and network bandwidth. So it is efficient to create one local vagrant box for DTaaS application and reuse the same in all installations. + +## Installed Software + +This base DTaaS vagrant box, when it is successfully created, has the following software: + +* docker +* nodejs and yarn +* jupyter +* microk8s +* containers + * mltooling/ml-workspace:0.13.2 + * traefik2.5 + * influxdb2.4 + * grafana + * telegraf + * gitlab + +## Create the vagrant box + +Publish a base virtualbox package to be used by +vagrant to publish all other virtualbox packages + +```bash +#create a key pair +ssh-keygen -b 4096 -t rsa -f key -q -N "" +mv key vagrant +mv key.pub vagrant.pub + +vagrant up + +# let the provisioning be complete +# replace the vagrant ssh key-pair with personal one +vagrant ssh + +# install the oh-my-zsh +sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" +# install plugins: history, autosuggestions, +git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions + +# inside ~/.zshrc, modify the following line +plugins=(git zsh-autosuggestions history cp tmux) + +# remove the vagrant default public key - first line of +# /home/vagrant/.ssh/authorized_keys + +# exit vagrant guest machine and then +# copy own private key to vagrant private key location +cp vagrant .vagrant/machines/default/virtualbox/private_key + +# check +vagrant ssh #should work + +vagrant halt + +vagrant package --base dtaas \ +--info "info.json" --output dtaas.vagrant + +# Add box to the vagrant cache in ~/.vagrant.d/boxes directory +vagrant box add --name dtaas ./dtaas.vagrant + +# You can use this box in other vagrant boxes using +#config.vm.box = "dtaas" +``` diff --git a/docs/admin/vagrant/single-machine.md b/docs/admin/vagrant/single-machine.md new file mode 100644 index 000000000..4e0ae427e --- /dev/null +++ b/docs/admin/vagrant/single-machine.md @@ -0,0 +1,30 @@ +# DTaaS on Single Vagrant Machine + +These are installation instructions for running DTaaS application inside one vagrant Virtual Machine. The setup requires a machine which can spare 16GB RAM, 8 vCPUs and 50GB Hard Disk space to the vagrant box. + +A dummy **foo.com** URL has been used for illustration. Please change this to your unique website URL. + +Please follow these steps to make this work in your local environment. + +1. Create [**dtaas** Vagrant box](../make_boxes/dtaas/README.md). You would have created an SSH key pair - _vagrant_ and _vagrant.pub_ for the vagrant box. The _vagrant_ is the private SSH key; Copy _vagrant_ SSH private key into the current directory (`deploy/vagrant/single-machine`). This shall be useful for logging into the vagrant machine created for single-machine deployment. +1. Update the **Vagrantfile**. Fields to update are: + 1. Hostname (`node.vm.hostname = "foo.com"`) + 1. MAC address (`:mac => "xxxxxxxx"`). This change is required if you have a DHCP server assigning domain names based on MAC address. Otherwise, you can leave this field unchanged. + 1. Other adjustments are optional. +1. Execute the following commands from terminal + +```bash +vagrant up +vagrant ssh +``` + +Set a cronjob inside the vagrant virtual machine to remote the conflicting default route. + +```bash +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh +sudo bash route.sh +``` + +If you only want to test the application and are not setting up a production instance, you can follow the instructions of [single script install](../trial.md). + +If you are not in a hurry and would rather have a production instance, follow the instructions of [regular server installation](../host.md) setup to complete the installation. diff --git a/docs/admin/vagrant/two-machines.md b/docs/admin/vagrant/two-machines.md new file mode 100644 index 000000000..70167af33 --- /dev/null +++ b/docs/admin/vagrant/two-machines.md @@ -0,0 +1,74 @@ +# DTaaS on Two Vagrant Machines + +These are installation instructions for running DTaaS application in two vagrant virtual machines (VMs). In this setup, all the user workspaces shall be run on server1 while all the platform services will be run on server2. + +The setup requires two server VMs with the following hardware configuration: + +**server1**: 16GB RAM, 8 x64 vCPUs and 50GB Hard Disk space + +**server2**: 6GB RAM, 3 x64 vCPUs and 50GB Hard Disk space + +Under the default configuration, two user workspaces are provisioned on server1. The default installation setup also installs InfluxDB, Grafana and RabbitMQ services on server2. If you would like to install more services, you can create shell scripts to install the same on server2. + +## Create Base Vagrant Box + +Create [**dtaas** Vagrant box](./base-box.md). You would have created an SSH key pair - _vagrant_ and _vagrant.pub_. The _vagrant_ is the private SSH key and is needed for the next steps. Copy _vagrant_ SSH private key into the current directory (`deploy/vagrant/single-machine`). This shall be useful for logging into the vagrant machines created for two-machine deployment. + +## Configure Server Settings + +**NOTE**: A dummy **foo.com** and **services.foo.com** URLs has been used for illustration. Please change these to your unique website URLs. + +The first step is to define the network identity of the two VMs. For that, you need _server name_, _hostname_ and _MAC address_. The hostname is the network URL at which the server can be accessed on the web. Please follow these steps to make this work in your local environment. + +Update the **boxes.json**. There are entries one for each server. The fields to update are: + + 1. `name` - name of server1 (`"name" = "dtaas"`) + 1. `hostname` - hostname of server1 (`"name" = "foo.com"`) + 1. MAC address (`:mac => "xxxxxxxx"`). This change is required if you have a DHCP server assigning domain names based on MAC address. Otherwise, you can leave this field unchanged. + 1. `name` - name of server2 (`"name" = "services"`) + 1. `hostname` - hostname of server2 (`"name" = "services.foo.com"`) + 1. MAC address (`:mac => "xxxxxxxx"`). This change is required if you have a DHCP server assigning domain names based on MAC address. Otherwise, you can leave this field unchanged. + 1. Other adjustments are optional. + +## Launch platform default services + +RabbitMQ, Grafana and InfluxDB services are provisioned on this server. +InfluxDB and visualization service will be available at: _services.foo.com_. The RabbitMQ service and its management interface shall be available at 5672 and 15672 TCP ports respectively. The Grafana service shall be available at TCP port 3000. + +The firewall and network access settings of corporate / cloud network need to be configured to allow external access to the services. Otherwise the users of DTaaS will not be able to utilize these services from their user workspaces. + +Execute the following commands from terminal to start the machine. + +```bash +vagrant up --provision services +vagrant ssh services +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/two-machine/services.sh +bash services.sh +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh +sudo bash route.sh +``` + +After the server is up and running, you can see the following services active within server2. + +| service | external url | +|:---|:---| +| InfluxDB and visualization service | services.foo.com | +| Grafana visualization service | services.foo.com:3000 | +| RabbitMQ communication service | services.foo.com:5672 | +| RabbitMQ management service | services.foo.com:15672 | +|| + +## Launch DTaaS application + +Execute the following commands from terminal + +```bash +vagrant up --provision dtaas +vagrant ssh dtaas +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh +sudo bash route.sh +``` + +If you only want to test the application and are not setting up a production instance, you can follow the instructions of [single script install](../trial.md). + +If you are not in a hurry and would rather have a production instance, follow the instructions of [regular server installation](../host.md) setup to complete the installation. diff --git a/docs/bugs.md b/docs/bugs.md index 19b5bb42d..1a8071923 100644 --- a/docs/bugs.md +++ b/docs/bugs.md @@ -1,4 +1,8 @@ +## Some limitations + +* The complete DTaaS software requires multiple docker containers and one client website. All of these can work together only on a server with a proper domain name. The complete application does not work on localhost. + ## Third-Party Software We use third-party software which have certain known issues. Some of the issues are listed below. @@ -8,3 +12,7 @@ We use third-party software which have certain known issues. Some of the issues - the docker container loses network connectivity after three days. The only known solution is to restart the docker container. You don't need to restart the complete DTaaS platform, restart of the docker contaienr of ml-workspace is sufficient. - the terminal tool doesn't seem to have the ability to refresh itself. If there is an issue, the only solution is to close and reopen the terminal from "open tools" drop down of notebook - terminal app does not show at all after some time: terminal always comes if it is open from drop-down menu of Jupyter Notebook, but not as a direct link. + +## Gitlab + +- The gilab oauth authentication service does not have a way to sign out of a third-party application. Even if you sign out of DTaaS, the gitlab still shows user as signed in. The next time you click on the sign in button on the DTaaS page, user is not shown the login page. Instead user is directly taken to the **Library** page. So close the brower window after you are done. Another way to overcome this limitation is to open your gitlab instance (`https://gitlab.foo.com`) and signout from there. Thus user needs to sign out of two places, namely DTaaS and gitlab, in order to completely exit the DTaaS application. diff --git a/docs/developer/client/uml/package-diagram.puml b/docs/developer/client/uml/package-diagram.puml index f57efc494..416d6cad3 100644 --- a/docs/developer/client/uml/package-diagram.puml +++ b/docs/developer/client/uml/package-diagram.puml @@ -75,4 +75,4 @@ Routes ..> Components : <> Util ..> Store Components ..> Util : <> Components ..> Store : <> -@enduml \ No newline at end of file +@enduml diff --git a/docs/developer/servers/lib/lib-class.puml b/docs/developer/servers/lib/lib-class.puml index 580fb66ab..a441ded30 100644 --- a/docs/developer/servers/lib/lib-class.puml +++ b/docs/developer/servers/lib/lib-class.puml @@ -44,4 +44,4 @@ FilesServiceFactory --> IFilesService: creates FilesServiceFactory --> ConfigService: uses GitlabFilesService --> ConfigService: uses LocalFilesService --> ConfigService: uses -@enduml \ No newline at end of file +@enduml diff --git a/docs/developer/servers/lib/lib-sequence.puml b/docs/developer/servers/lib/lib-sequence.puml index a30110611..84cd5e9cd 100644 --- a/docs/developer/servers/lib/lib-sequence.puml +++ b/docs/developer/servers/lib/lib-sequence.puml @@ -95,4 +95,4 @@ else Successful operation end deactivate FR -@enduml \ No newline at end of file +@enduml diff --git a/docs/developer/system/DTaaS-overview.pdf b/docs/developer/system/DTaaS-overview.pdf deleted file mode 100755 index 1ba28bbca..000000000 Binary files a/docs/developer/system/DTaaS-overview.pdf and /dev/null differ diff --git a/docs/developer/system/DTaaS.drawio b/docs/developer/system/DTaaS.drawio old mode 100755 new mode 100644 diff --git a/docs/index.md b/docs/index.md index d0fc7a62e..91e113f43 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,6 +1,11 @@ The Digital Twin as a Service (DTaaS) software is useful to create and run digital twins. The digital twins that are running can be used as service by other users. These users need not be members of the DTaaS software platform itself. -A brief overview of the software is available in [this presentation](/developer/system/DTaaS-overview.pdf) and [recorded video](https://www.dropbox.com/s/mgxxf5chp9b130x/DTaaS%20presentation%20and%20brainstorming-20230317.mp4?dl=1). There is also a [research paper](https://arxiv.org/abs/2305.07244) if you are interested in reading the scientific roadmap for this software. +There is an overview of the software available for: + +* General users - [overview slides](https://odin.cps.digit.au.dk/into-cps/dtaas/assets/DTaaS-short-intro.pdf) and [overview video](https://odin.cps.digit.au.dk/into-cps/dtaas/assets/videos/DTaaS-short-intro.mp4), [feature walkthrough](https://odin.cps.digit.au.dk/into-cps/dtaas/assets/videos/dtaas-v0.2.0-demo.mp4) +* Developers - [slides](https://odin.cps.digit.au.dk/into-cps/dtaas/assets/DTaaS-overview.pdf) and [video](https://odin.cps.digit.au.dk/into-cps/dtaas/assets/videos/DTaaS-overview.mkv). + +There is also a [research paper draft](https://arxiv.org/abs/2305.07244) if you are interested in reading the scientific roadmap for this software. ## License diff --git a/docs/redirect-page.html b/docs/redirect-page.html index f4bc9c66f..b7b4bd918 100644 --- a/docs/redirect-page.html +++ b/docs/redirect-page.html @@ -1,15 +1,71 @@ - - -Digital Twin as a Service - - - Please select the version of the documents: - -
    -
  1. latest: online offline
    2. -
  2. version0.2: online offline
    3. -
- - - \ No newline at end of file + + + + + + + Digital Twin as a Service + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Digital Twin as a Service (DTaaS)
Documentation
Version
DevelopmentOnlinePDF
Version 0.2.0OnlinePDF
+ + diff --git a/docs/requirements-github.txt b/docs/requirements-github.txt new file mode 100644 index 000000000..146bf2e03 --- /dev/null +++ b/docs/requirements-github.txt @@ -0,0 +1,4 @@ +mkdocs +mkdocs-material +python-markdown-math +mkdocs-open-in-new-tab diff --git a/docs/requirements.txt b/docs/requirements.txt index 876f7d935..35d867370 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1,3 +1,6 @@ mkdocs mkdocs-material -python-markdown-math \ No newline at end of file +python-markdown-math +mkdocs-open-in-new-tab +mkdocs-with-pdf +qrcode \ No newline at end of file diff --git a/docs/thanks.md b/docs/thanks.md index 30a715b7c..5983ad686 100644 --- a/docs/thanks.md +++ b/docs/thanks.md @@ -5,8 +5,9 @@ ## Users -Cláudio Ângelo Gonçalves Gomes, Hao Feng, Tanusree Roy +Cláudio Ângelo Gonçalves Gomes, Dmitri Tcherniak, Elif Ecem Bas, Giuseppe Abbiati, Hao Feng, Henrik Ejersbo, Tanusree Roy ## Documentation -1. Talasila, P., Gomes, C., Mikkelsen, P. H., Arboleda, S. G., Kamburjan, E., & Larsen, P. G. (2023). [Digital Twin as a Service (DTaaS): A Platform for Digital Twin Developers and Users](https://arxiv.org/abs/2305.07244). arXiv preprint arXiv:2305.07244. \ No newline at end of file +1. Talasila, P., Gomes, C., Mikkelsen, P. H., Arboleda, S. G., Kamburjan, E., & Larsen, P. G. (2023). [Digital Twin as a Service (DTaaS): A Platform for Digital Twin Developers and Users](https://arxiv.org/abs/2305.07244). arXiv preprint arXiv:2305.07244. +2. Astitva Sehgal diff --git a/docs/user/servers/lib/LIB-MS.md b/docs/user/servers/lib/LIB-MS.md index c1f524901..41f2db931 100644 --- a/docs/user/servers/lib/LIB-MS.md +++ b/docs/user/servers/lib/LIB-MS.md @@ -248,7 +248,7 @@ send requests to: https://foo.com/lib } } } - ``` + ``` === "HTTP Request :warning:" diff --git a/mkdocs_offline.yml b/mkdocs-github.yml similarity index 77% rename from mkdocs_offline.yml rename to mkdocs-github.yml index 7cf5db10f..cdf6843cc 100644 --- a/mkdocs_offline.yml +++ b/mkdocs-github.yml @@ -14,14 +14,19 @@ theme: - navigation.tracking - navigation.footer -plugins: - - offline - nav: - Home: index.md - Admin: - - website: admin/client/CLIENT.md - - library microservice: admin/servers/lib/LIB-MS.md + - Host Install: + - Trial: admin/trial.md + - Production: admin/host.md + - Vagrant: + - Base Box: admin/vagrant/base-box.md + - Single Machine: admin/vagrant/single-machine.md + - Two Machines: admin/vagrant/two-machines.md + - Separate Packages: + - website: admin/client/CLIENT.md + - library microservice: admin/servers/lib/LIB-MS.md - User: - Features: user/features.md - Website: user/website/index.md @@ -63,4 +68,8 @@ markdown_extensions: - pymdownx.tabbed: alternate_style: true - mdx_math: - enable_dollar_delimiter: True \ No newline at end of file + enable_dollar_delimiter: True + +plugins: + - open-in-new-tab + - search diff --git a/mkdocs.yml b/mkdocs.yml old mode 100644 new mode 100755 index 702e1bd69..fc9935c26 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -17,8 +17,16 @@ theme: nav: - Home: index.md - Admin: - - website: admin/client/CLIENT.md - - library microservice: admin/servers/lib/LIB-MS.md + - Host Install: + - Trial: admin/trial.md + - Production: admin/host.md + - Vagrant: + - Base Box: admin/vagrant/base-box.md + - Single Machine: admin/vagrant/single-machine.md + - Two Machines: admin/vagrant/two-machines.md + - Separate Packages: + - website: admin/client/CLIENT.md + - library microservice: admin/servers/lib/LIB-MS.md - User: - Features: user/features.md - Website: user/website/index.md @@ -60,4 +68,15 @@ markdown_extensions: - pymdownx.tabbed: alternate_style: true - mdx_math: - enable_dollar_delimiter: True \ No newline at end of file + enable_dollar_delimiter: True + +plugins: + - open-in-new-tab + - search + - with-pdf: + author: "DTaaS Development Team" + cover: true + cover_logo: assets/into-cps-logo.png + cover_subtitle: "" + enabled_if_env: MKDOCS_ENABLE_PDF_EXPORT + output_path: pdf/DTaaS-docs.pdf diff --git a/script/docker.sh b/script/docker.sh new file mode 100644 index 000000000..ffa5bd2ff --- /dev/null +++ b/script/docker.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# get the required docker images +docker pull traefik:v2.5 +docker pull influxdb:2.4 +docker pull mltooling/ml-workspace:0.13.2 +docker pull grafana/grafana +docker pull telegraf +docker pull gitlab/gitlab-ce:15.10.0-ce.0 diff --git a/script/docs.sh b/script/docs.sh index 3ce886cb5..252ec3ec3 100755 --- a/script/docs.sh +++ b/script/docs.sh @@ -3,7 +3,7 @@ if [ -n "$1" ]; then VERSION="$1" else - VERSION="latest" + VERSION="development" fi export VERSION @@ -11,17 +11,19 @@ TOP_DIR=$(pwd) export TOP_DIR COMMIT_HASH=$(git rev-parse --short HEAD) export COMMIT_HASH +export MKDOCS_ENABLE_PDF_EXPORT=1 echo ${VERSION} +if [ -d site ] +then + rm -rf site +fi + printf "generate and publish documents" mkdocs build --config-file mkdocs.yml --site-dir "site/online/${VERSION}" -mkdocs build --config-file mkdocs_offline.yml --site-dir "site/offline/${VERSION}" cp docs/redirect-page.html site/index.html -cd site/offline || exit -zip -r latest.zip latest - cd "${TOP_DIR}" || exit git checkout webpage-docs @@ -31,10 +33,21 @@ git checkout webpage-docs # rm -rf .git-hooks/* # rm script/configure-git-hooks.sh script/grafana.sh script/influx.sh script/install.bash -rm -rf latest -mv site/online/latest . -mv site/offline/latest.zip . +cd "${TOP_DIR}" || exit +if [ -d "${VERSION}" ]; then + rm -rf "${VERSION}" +fi + +if [ -d "site/online/${VERSION}" ] +then + cd site/online || exit + mv "${VERSION}/pdf/DTaaS-docs.pdf" "${TOP_DIR}/DTaaS-${VERSION}.pdf" + mv "${VERSION}" "${TOP_DIR}/." +fi + +cd "${TOP_DIR}" || exit mv site/index.html . +rm -rf site git add . git commit -m "docs for ${COMMIT_HASH} commit" \ No newline at end of file diff --git a/script/env.sh b/script/env.sh new file mode 100755 index 000000000..90301a022 --- /dev/null +++ b/script/env.sh @@ -0,0 +1,84 @@ +#!/bin/bash +sudo apt-get update -y +sudo apt-get upgrade -y + + +# Install docker for containers and microservices +# https://docs.docker.com/engine/install/ubuntu/ +sudo apt-get install -y \ + ca-certificates \ + curl \ + gnupg \ + lsb-release \ + zsh \ + apache2-utils \ + net-tools + +sudo mkdir -p /etc/apt/keyrings +if [ ! -f /etc/apt/keyrings/docker.gpg ] +then + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg + sudo printf \ + "deb [arch=%s signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ + %s stable" "$(dpkg --print-architecture)" "$(lsb_release -cs)" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null +fi + +sudo apt-get update -y +sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin +sudo groupadd docker || true +sudo usermod -aG docker "$USER" || true + +printf "\n\n\nMake docker available to your user account....\n" +printf ".......\n" +printf "Your user account is member of:\n" +groups +printf "groups.\n" + +printf "If your user account is a member of docker group, let the installation script continue.\n" +printf "Otherwise, exit this script and run the following command\n\n" +printf "sudo usermod -aG docker %s\n\n" "$USER" +printf "logout and login again. You can run this script again after login\n\n" +printf "Press Ctl+C if you need to complete the this task....\n" +printf "Waiting for 60 seconds....\n" +sleep 60 + + +#newgrp docker +sudo service docker start +docker run hello-world + +sudo systemctl enable docker.service +sudo systemctl enable containerd.service + + + +# Install nodejs environment +curl -fsSL https://deb.nodesource.com/setup_18.x | sudo bash - +sudo apt-get install -y nodejs + +if [ ! -f /usr/share/keyrings/yarnkey.gpg ] +then + curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | gpg --dearmor | sudo tee /usr/share/keyrings/yarnkey.gpg >/dev/null + printf "deb [signed-by=/usr/share/keyrings/yarnkey.gpg] https://dl.yarnpkg.com/debian stable main" | sudo tee /etc/apt/sources.list.d/yarn.list +fi + +sudo apt-get update -y +sudo apt-get install -y yarn +sudo npm install -g serve + +# Install openssl for certificate generation +sudo apt-get install -y wget openssl + + +# Install playwright tool for integration tests on browsers +npx playwright install-deps + +#------------- +printf "\n\n Installing required python packages...." +apt install -y python3-pip +sudo -H pip3 install mkdocs +sudo -H pip3 install mkdocs-material +sudo -H pip3 install python-markdown-math +sudo -H pip3 install mkdocs-open-in-new-tab +sudo -H pip3 install mkdocs-with-pdf +sudo -H pip3 install qrcode diff --git a/script/grafana.sh b/script/grafana.sh deleted file mode 100755 index 177b4df27..000000000 --- a/script/grafana.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/bin/bash -printf "Grafana provisioning script\n" -# Command to launch grafana - -docker run -d \ - -p 3000:3000 \ - --name=grafana-test \ - -e "GF_SERVER_SERVE_FROM_SUB_PATH=true" \ - -e "GF_SERVER_DOMAIN=localhost" \ - -e "GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/vis" \ - -e "GF_AUTH_BASIC_ENABLED=false" \ - -e "GF_AUTH_PROXY_ENABLED=false" \ - -e "GF_SECURITY_ALLOW_EMBEDDING=true" \ - -e "GF_AUTH_ANONYMOUS_ENABLED=true" \ - -e "GF_AUTH_ANONYMOUS_ORG_NAME=Main" \ - -e "GF_AUTH_ANONYMOUS_ORG_ROLE=Editor" \ - -e "GF_USERS_ALLOW_SIGN_UP=false" \ - -e "GF_FEATURE_TOGGLES_ENABLE=publicDashboards" \ - -e "GF_PATHS_CONFIG=/etc/grafana/grafana.ini" \ - -e "GF_PATHS_DATA=/var/lib/grafana" \ - -e "GF_PATHS_HOME=/usr/share/grafana" \ - -e "GF_PATHS_LOGS=/var/log/grafana" \ - -e "GF_PATHS_PLUGINS=/var/lib/grafana/plugins" \ - -e "GF_PATHS_PROVISIONING=/etc/grafana/provisioning" \ - -e "HOME=/home/grafana" \ - grafana/grafana -printf "Complete the setup from GUI" - diff --git a/script/influx.sh b/script/influx.sh deleted file mode 100755 index 58d4b912a..000000000 --- a/script/influx.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/bash -printf "InfluxDB provision script" - - -#------------- -printf "\n\n start the InfluxDB server" -printf ".........................." -# note: InfluxDB doesn't work on /vagrant shared folder -INFLUXDB_DATA="${PWD}/data/influxdb2" -mkdir -p "$INFLUXDB_DATA" - -# Remember to change the settings -# src: https://hub.docker.com/_/influxdb/ -docker run -d -p 80:8086 \ - --name influxdb24 \ - -v "$INFLUXDB_DATA/data":/var/lib/influxdb2 \ - -v "$INFLUXDB_DATA/config":/etc/influxdb2 \ - -e DOCKER_INFLUXDB_INIT_MODE=setup \ - -e DOCKER_INFLUXDB_INIT_USERNAME=dtaas \ - -e DOCKER_INFLUXDB_INIT_PASSWORD=dtaas1357 \ - -e DOCKER_INFLUXDB_INIT_ORG=dtaas \ - -e DOCKER_INFLUXDB_INIT_BUCKET=dtaas \ - influxdb:2.4 - -#docker run -d -p 9086:8086 \ -# --name influx24 \ -# -v ${PWD}/data/influxdb2:/var/lib/influxdb2 \ -# influxdb:2.4 -#printf "Complete the setup from GUI" - diff --git a/script/install.bash b/script/install.bash deleted file mode 100755 index 0e3b8010a..000000000 --- a/script/install.bash +++ /dev/null @@ -1,64 +0,0 @@ -#!/bin/bash -apt-get update -y -apt-get upgrade -y - - -# Install docker for containers and microservices -# https://docs.docker.com/engine/install/ubuntu/ -apt-get install -y \ - ca-certificates \ - curl \ - gnupg \ - lsb-release \ - zsh \ - apache2-utils - -mkdir -p /etc/apt/keyrings -curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg -printf \ - "deb [arch=%s signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ - %s stable" "$(dpkg --print-architecture)" "$(lsb_release -cs)" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null -apt-get update -y -apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin -groupadd docker -usermod -aG docker "$USER" -#newgrp docker -service docker start -docker run hello-world - -systemctl enable docker.service -systemctl enable containerd.service - - - -# Install nodejs environment -curl -fsSL https://deb.nodesource.com/setup_18.x | bash - -apt-get install -y nodejs -curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | gpg --dearmor | sudo tee /usr/share/keyrings/yarnkey.gpg >/dev/null -printf "deb [signed-by=/usr/share/keyrings/yarnkey.gpg] https://dl.yarnpkg.com/debian stable main" | sudo tee /etc/apt/sources.list.d/yarn.list -apt-get update -y -apt-get install -y yarn -npm install -g serve - -# Install openssl for certificate generation -apt-get install -y wget openssl - - -# Install jupter toolchain -apt install -y python3-pip -sudo -H pip install jupyterlab - -# Install playwright tool for integration tests on browsers -npx playwright install-deps - -#------------- -printf "\n\n Install jupyter toolchain" -apt install -y python3-pip -sudo -H pip install jupyterlab -sudo -H pip install mkdocs -sudo -H pip3 install mkdocs-material -sudo -H pip3 install python-markdown-math - -#install docker-compose from https://docs.docker.com/compose/install/other/ -curl -SL "https://github.com/docker/compose/releases/download/v2.15.1/docker-compose-linux-x86_64" -o /usr/local/bin/docker-compose -ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose diff --git a/servers/config/gateway/README.md b/servers/config/gateway/README.md index df274f524..8d26efa1b 100644 --- a/servers/config/gateway/README.md +++ b/servers/config/gateway/README.md @@ -14,6 +14,7 @@ The default configuration uses two services at the following URLs: | localhost | React Website | localhost:4000 | | localhost/lib | Lib Microservice | localhost:4001 | | localhost/user1 | ML Workspace | localhost:8090 | +| localhost/user2 | ML Workspace | localhost:8091 | | | ## Start the Gateway @@ -27,10 +28,6 @@ docker run -d \ traefik:v2.5 ``` -## Access - -# Run Lib MS at port 4001 - ## Authentication The dummy username is `foo` and the password is `bar`. @@ -59,4 +56,5 @@ The routes / URLs need to be updated for your local setup. The current version o | foo.com | React Website | localhost:4000 | | foo.com/lib | Lib Microservice | localhost:4001 | | foo.com/user1 | ML Workspace | localhost:8090 | +| foo.com/user2 | ML Workspace | localhost:8091 | | | diff --git a/servers/lib/README.md b/servers/lib/README.md index ead772a5c..754d61ce2 100644 --- a/servers/lib/README.md +++ b/servers/lib/README.md @@ -1,15 +1,22 @@ # Overview -This **lib microservice** takes a query from the user that asks for content within a given specified directory. This microservices handles request by fetching and returning the file-names and folders within that respective directory. This microservice provides gitlab graphql compliant API. +The **lib microservice** is a simplified file manager providing graphQL API. It has three features: + +* provide a listing of directory contents. +* transfer a file to user. +* Source files can either come from local file system or from a gitlab instance. ## Gitlab setup -For this microserivce to be functional, a certain directory or gitlab project structure is expected. The microservice expects that the gitlab consisting of one group, DTaaS, and within that group, all of the projects be located, **user1**, **user2**, ... , as well as a **commons** project. A sample file structure can be seen in `files/` directory. +For this microserivce to be functional, a certain directory or gitlab project structure is expected. The microservice expects that the gitlab consisting of one group, DTaaS, and within that group, all of the projects be located, **user1**, **user2**, ... , as well as a **commons** project. Each project corresponds to files of one user. +A sample file structure can be seen in [gitlab dtaas group](https://gitlab.com/dtaas). You can visit the gitlab documentation on [groups](https://docs.gitlab.com/ee/user/group/) for help on the management of gitlab groups. -On how to create groups, visit the [gitlab documentation](https://docs.gitlab.com/ee/user/group/) +You can clone the git repositories from the `dtaas` group to get a sample file system structure for the lib microservice. ## Configuration setup +The microservices uses `.env` environment files to receive configuration. + In order to create this environment, you need to create a `.env` file, wherein you create the following environment variables, and insert with the correct-information relevant for your setup: @@ -21,21 +28,31 @@ GITLAB_GROUP ='dtaas' GITLAB_URL='https://gitlab.com/api/graphql' TOKEN='123-sample-token' LOG_LEVEL='debug' -TEST_PATH='/Users//DTaaS/servers/lib/test/data/test_assets' APOLLO_PATH='/lib' or '' GRAPHQL_PLAYGROUND='false' or 'true' - ``` -The `TOKEN` should be set to your GitLab Group access API token. For more information on how to create and use your access token, visit: -https://docs.gitlab.com/ee/user/group/settings/group_access_tokens.html +The `TOKEN` should be set to your GitLab Group access API token. For more information on how to create and use your access token, [gitlab page](https://docs.gitlab.com/ee/user/group/settings/group_access_tokens.html). Once you've generated a token, copy it and replace the value of `TOKEN` with your token for the gitlab group, can be found. +## User Commands + +```bash +yarn install # Install dependencies for the microservice +yarn build # build the application +yarn start # start the application +``` + +You can press `Ctl+C` to halt the application. If you wish to run the microservice in the background, use + +```bash +nohup yarn start & disown +``` + ## Developer Commands ```bash -cd server/lib yarn install # Install dependencies for the microservice yarn syntax # analyzes source code for potential errors, style violations, and other issues, yarn build # compile ES6 files into ES5 javascript files and copy all JS files into build/ directory @@ -51,89 +68,133 @@ yarn clean # deletes directories "build", "coverage", and "dist" The URL endpoint for this microservice is located at: `localhost:PORT/lib` -### Lib request and response +## GraphQL API Calls -Documentation of the lib query and responses. +The lib microservice takes two distinct GraphQL queries. -The provided HTTP request is a POST request sent to the endpoint http://foo.com:/lib. It contains a JSON payload in the body. The request includes headers such as Content-Type, User-Agent, and Accept. +### Directory Listing -The request is using a POST method because it is sending a GraphQL query. GraphQL queries are usually sent as POST requests because they can be complex and include a JSON payload, allowing for more flexibility in the query structure and variables +This query receives directory path and provides list of files in that directory. A sample query and response are given here. +``` graphql +query { + listDirectory(path: "user1") { + repository { + tree { + blobs { + edges { + node { + name + type + } + } + } + trees { + edges { + node { + name + type + } + } + } + } + } + } +} ``` -HTTP Request: - -..... - -send the request to: http://foo.com:/lib - -POST /lib - -Host: foo.com: - -Content-Type:application/json - -User-Agent:Mozilla - -Accept:*/* - +``` graphql { - -"query": "{ getFiles(path: \"common\")}" - + "data": { + "listDirectory": { + "repository": { + "tree": { + "blobs": { + "edges": [] + }, + "trees": { + "edges": [ + { + "node": { + "name": "common", + "type": "tree" + } + }, + { + "node": { + "name": "data", + "type": "tree" + } + }, + { + "node": { + "name": "digital twins", + "type": "tree" + } + }, + { + "node": { + "name": "functions", + "type": "tree" + } + }, + { + "node": { + "name": "models", + "type": "tree" + } + }, + { + "node": { + "name": "tools", + "type": "tree" + } + } + ] + } + } + } + } + } } - -HTTP Response: - -..... - -200 OK - -access-control-allow-origin: * - -connection: keep-alive - -content-length: 76 - -content-type: application/json; charset=utf-8 - -date: Mon, 15 May 2023 10:13:37 GMT - -etag: ................ - -keep-alive: timeout=5 - -x-powered-by: Express - -{'data':{'getFiles':['data','digital twins','functions','models','tools']}} - ``` -### GraphQL API queries +### Fetch a file -The only accepted query is: +This query receives directory path and send the file contents to user in response. A sample query and response are given here. ```graphql -query directoryList($path: String!, $domain: ID!) { - project(fullPath: $domain) { - webUrl - path +query { + readFile(path: "user2/data/sample.txt") { repository { - paginatedTree(path: $path, recursive: false) { + blobs { nodes { - trees { - nodes { - name - } - } + name + rawBlob + rawTextBlob } } - diskPath } } } ``` -The _path_ refers to the file path to look at: For example, _user1_ looks at files of **user1**; _user1/functions_ looks at contents of _functions/_ directory. - -The _$domain_ refers to gitlab project to lookup. This information is ignored by the microservice. Instead, the microservice takes the _$domain_ variable value from `.env` file. +```graphql +{ + "data": { + "readFile": { + "repository": { + "blobs": { + "nodes": [ + { + "name": "sample.txt", + "rawBlob": "hello world", + "rawTextBlob": "hello world" + } + ] + } + } + } + } +} +``` diff --git a/ssl/certificates.bash b/ssl/certificates.bash old mode 100755 new mode 100644