Feature/distributed demo

.github/workflows/docs.yml

@@ -22,8 +22,8 @@ jobs:
         uses: Tiryoh/actions-mkdocs@v0
         with:
           mkdocs_version: 'latest'
-          requirements: 'docs/requirements.txt'
-          configfile: 'mkdocs.yml'
+          requirements: 'docs/requirements-github.txt'
+          configfile: 'mkdocs-github.yml'
 
 
       - name: show contents of compiled docs

README.md

@@ -28,13 +28,21 @@ The rest of the information on this page is aimed at current and potential contr
 To install the development environment, run
 
 ```bash
-bash script/install.bash
+bash script/env.sh
 ```
 
+There is a script to download all the docker containers used in the project. You can download them using
+
+```bash
+bash script/docker.sh
+```
+
+**CAVEAT**: The docker images are large and are likely to consume about 5GB of bandwidth and 15GB of space. You will have to download the docker images on a really good network.
+
 Before you make commits, please install the git hooks provided in the repository.
 
-```shell
-script/configure-git-hooks.sh
+```bash
+bash script/configure-git-hooks.sh
 ```
 
 This will ensure that your commits are formatted correctly and that the unittests pass before you push your changes. Be aware that the tests take a long time to run. If you want to skip the tests or formatting, you can use the `--no-verify` flag on `git commit` or `git push`.

STATUS.md

@@ -7,3 +7,11 @@
 [![Client CI](https://github.com/into-cps-association/dtaas/actions/workflows/client.yml/badge.svg)](https://github.com/into-cps-association/dtaas/actions/workflows/client.yml/badge.svg)
 
 [![Server CI](https://github.com/into-cps-association/dtaas/actions/workflows/server.yml/badge.svg)](https://github.com/into-cps-association/dtaas/actions/workflows/server.yml/badge.svg)
+
+[![Contributors](https://img.shields.io/github/contributors/INTO-CPS-Association/DTaaS)](https://github.com/INTO-CPS-Association/DTaaS/graphs/contributors)
+
+[![Latest Commit](https://img.shields.io/github/last-commit/INTO-CPS-Association/DTaaS)](https://github.com/INTO-CPS-Association/DTaaS/commits/feature/distributed-demo)
+
+[![INTO-CPS Association](https://img.shields.io/badge/INTO_CPS_Association-white)](https://into-cps.org/)
+
+

client/config/prod.js

@@ -1,5 +1,5 @@
 window.env = {
-  REACT_APP_ENVIRONMENT: 'dev',
+  REACT_APP_ENVIRONMENT: 'prod',
   REACT_APP_URL: 'https://foo.com/',
   REACT_APP_URL_BASENAME: 'dtaas',
   REACT_APP_URL_DTLINK: '/lab',
@@ -15,4 +15,4 @@ window.env = {
   REACT_APP_REDIRECT_URI: 'https://foo.com/Library',
   REACT_APP_LOGOUT_REDIRECT_URI: 'https://foo.com/',
   REACT_APP_GITLAB_SCOPES: 'openid profile read_user read_repository api',
-};
+};

deploy/README.md

@@ -0,0 +1,93 @@
+# DTaaS on Linux Operating System
+
+This directory contains code for running DTaaS application on a Ubuntu Server 20.04 Operating System. The setup requires a machine which can spare 16GB RAM, 8 vCPUs and 50GB Hard Disk space.
+
+A dummy **foo.com** URL has been used for illustration. Please change this to your unique website URL. It is assumed that you are going to serve the application in only HTTPS mode.
+
+Please follow these steps to make this work in your local environment. Download the codebase as zip file into your computer and unzip the same into a directory named **DTaaS**. The rest of the instructions assume that your working directory is **DTaaS**.
+
+## Configuration
+
+You need to configure the gateway, library microservice and react client website.
+
+The first step is to decide on the number of users and their usenames. The traefik gateway configuration has a template for two users. You can modify the usernames in the template to the usernames chosen by you.
+
+### The traefik gateway server
+
+You can run the Run the Traefik gateway server in both and HTTPS and HTTPS mode to experience the DTaaS application. The installation guide assumes that you can run the application in HTTPS mode.
+
+The Traefik gateway configuration is at [fileConfig](../config/gateway/fileConfig.yml). Change `localhost` to `foo.com` and user1/user2 to the usernames chosen by you. 
+
+**NOTE**: Do not use `http://` or `https://` in [fileConfig](../config/gateway/fileConfig.yml).
+
+#### Authentication
+
+The dummy username is `foo` and the password is `bar`.
+Please change this before starting the gateway.
+
+```bash
+rm deploy/config/gateway/auth
+touch deploy/config/gateway/auth
+htpasswd deploy/config/gateway/auth <first_username>
+password: <your password>
+```
+
+The user credentials added in [auth](../config/gateway/auth) should match the usernames in [fileConfig](../config/gateway/fileConfig.yml).
+
+## Configure lib microservice
+The library microservice requires configuration. A template of this configuration file is given in _config/lib_ file. Please modify this file as per your needs. 
+
+The first step in this configuration is to prepare the a filesystem for users. An example file system in `files/` directory. You can rename the top-level user1/user2 to the usernames chosen by you.
+
+Add an environment file named .env in lib for the library microservice. An example `.env` file is given below. The simplest possibility is to use `local` mode with the following example. The filepath is the absolute filepath to `files/` directory. You can copy this configuration into _config/lib_ file to get started.
+```env
+PORT='4001'
+MODE='local'
+LOCAL_PATH ='filepath'
+LOG_LEVEL='debug'
+APOLLO_PATH='/lib'
+GRAPHQL_PLAYGROUND='true'
+```
+
+## Configure react website
+
+Change the React website configuration in _deploy/config/client/env.js_.
+
+```js
+window.env = {
+  REACT_APP_ENVIRONMENT: 'prod',
+  REACT_APP_URL: 'https://foo.com/',
+  REACT_APP_URL_BASENAME: 'dtaas',
+  REACT_APP_URL_DTLINK: '/lab',
+  REACT_APP_URL_LIBLINK: '',
+  REACT_APP_WORKBENCHLINK_TERMINAL: '/terminals/main',
+  REACT_APP_WORKBENCHLINK_VNCDESKTOP: '/tools/vnc/?password=vncpassword',
+  REACT_APP_WORKBENCHLINK_VSCODE: '/tools/vscode/',
+  REACT_APP_WORKBENCHLINK_JUPYTERLAB: '/lab',
+  REACT_APP_WORKBENCHLINK_JUPYTERNOTEBOOK: '',
+
+  REACT_APP_CLIENT_ID: '934b98f03f1b6f743832b2840bf7cccaed93c3bfe579093dd0942a433691ccc0',
+  REACT_APP_AUTH_AUTHORITY: 'https://gitlab.foo.com/',
+  REACT_APP_REDIRECT_URI: 'https://foo.com/Library',
+  REACT_APP_LOGOUT_REDIRECT_URI: 'https://foo.com/',
+  REACT_APP_GITLAB_SCOPES: 'openid profile read_user read_repository api',
+};
+```
+
+## Update the installation script
+
+Open `deploy/install.sh` and update user1/user2 to usernames chosen by you.
+
+## Perform the Installation
+
+Go to the DTaaS directory and execute
+
+```sh
+source deploy/install.sh
+```
+
+You can run this script multiple times until the installation is successful.
+
+## Access the application
+
+Now you should be able to access the DTaaS application at: _https://foo.com_

deploy/config/client/env.js

@@ -0,0 +1,18 @@
+window.env = {
+  REACT_APP_ENVIRONMENT: 'dev',
+  REACT_APP_URL: 'https://foo.com/',
+  REACT_APP_URL_BASENAME: 'dtaas',
+  REACT_APP_URL_DTLINK: '/lab',
+  REACT_APP_URL_LIBLINK: '',
+  REACT_APP_WORKBENCHLINK_TERMINAL: '/terminals/main',
+  REACT_APP_WORKBENCHLINK_VNCDESKTOP: '/tools/vnc/?password=vncpassword',
+  REACT_APP_WORKBENCHLINK_VSCODE: '/tools/vscode/',
+  REACT_APP_WORKBENCHLINK_JUPYTERLAB: '/lab',
+  REACT_APP_WORKBENCHLINK_JUPYTERNOTEBOOK: '',
+
+  REACT_APP_CLIENT_ID: '934b98f03f1b6f743832b2840bf7cccaed93c3bfe579093dd0942a433691ccc0',
+  REACT_APP_AUTH_AUTHORITY: 'https://gitlab.foo.com/',
+  REACT_APP_REDIRECT_URI: 'https://foo.com/Library',
+  REACT_APP_LOGOUT_REDIRECT_URI: 'https://foo.com/',
+  REACT_APP_GITLAB_SCOPES: 'openid profile read_user read_repository api',
+};

deploy/config/gateway/auth

@@ -0,0 +1 @@
+foo:$apr1$fGdMVLcl$dMzJW.JF8Rn6Vzrf5uxaU/

deploy/config/lib

@@ -0,0 +1,9 @@
+PORT='4001'
+MODE='local' or 'gitlab'
+LOCAL_PATH ='filepath'
+GITLAB_GROUP ='dtaas'
+GITLAB_URL='https://gitlab.foo.com/api/graphql'
+TOKEN='123-sample-token'
+LOG_LEVEL='debug'
+APOLLO_PATH='/lib' or ''
+GRAPHQL_PLAYGROUND='false' or 'true'

deploy/create-cronjob.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+# Add a cronjob for the DTaaS services to the current users crontab
+# the actual crontab script is cron.sh
+# all existing crontab jobs are retained
+
+if [ -n "$1" ]; then
+  PROJECT_PATH="$1"
+else
+  PROJECT_PATH="$(pwd)"
+fi
+
+export PROJECT_PATH
+
+printf "Existing crontab for the user\n"
+crontab -l
+crontab -l | sort -u  > temp.cron
+echo "* * * * * ${PROJECT_PATH}/deploy/cron.sh ${PROJECT_PATH}" >> temp.cron
+sort -u temp.cron > jobs.cron
+crontab jobs.cron
+rm jobs.cron temp.cron
+printf "Updated crontab for the user\n"
+crontab -l

deploy/cron.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+# Run the client website and lib microservice in background mode
+
+PROJECT_PATH="$1"
+
+nc -z localhost 4000
+PORT_STATUS=$?
+if (( PORT_STATUS == 1 ))
+then
+  echo "starting react website"
+  cd "${PROJECT_PATH}/client" || exit
+  nohup serve -s build -l 4000 & disown
+fi
+
+
+
+nc -z localhost 4001
+PORT_STATUS=$?
+if (( PORT_STATUS == 1 ))
+then
+  cd "${PROJECT_PATH}/servers/lib" || exit
+  nohup yarn start & disown
+fi
+
+docker start traefik-gateway

deploy/install.sh

@@ -0,0 +1,94 @@
+#!/bin/bash
+set -eu
+
+printf "NOTE\n "
+printf "....\n "
+printf "This script installs DTaaS application for two users.\n "
+printf "Remember to configure the application before proceeding further....\n "
+printf "All the configuration instructions are in deploy/README.md file.\n "
+printf ".........\n \n \n "
+
+printf "Press Ctl+C if you need to complete the configuration....\n "
+printf "Waiting for 60 seconds....\n "
+sleep 60
+
+printf "\n \n Install the system dependencies...\n "
+printf "....\n "
+bash script/env.sh || exit
+
+printf "\n \n Download the required docker images...\n "
+printf ".........\n "
+source script/docker.sh || exit
+
+
+printf "\n \n Continue with the DTaaS installation...\n "
+printf ".........\n "
+TOP_DIR=$(pwd)
+
+printf "\n \n Build, configure and run the react website\n "
+printf ".....\n "
+cd "${TOP_DIR}/client" || exit
+yarn install
+yarn build
+
+#one of the environments; specify only one; "dev" used the REACT_APP_ENV is not set
+yarn configapp prod
+cp "${TOP_DIR}/deploy/config/client/env.js" build/env.js
+nohup serve -s build -l 4000 & disown
+
+#-------------
+printf "\n\nStart the lib microservice\n "
+printf "...........\n "
+cd "${TOP_DIR}/servers/lib" || exit
+yarn install
+yarn build
+cp "${TOP_DIR}/deploy/config/lib" .env
+nohup yarn start & disown
+
+#-------------
+printf "\n \n Start the user workspaces\n "
+printf "...........\n "
+docker run -d \
+ -p 8090:8080 \
+  --name "ml-workspace-user1" \
+  -v "${TOP_DIR}/files/user1:/workspace" \
+  -v "${TOP_DIR}/files/common:/workspace/common:ro" \
+  --env AUTHENTICATE_VIA_JUPYTER="" \
+  --env WORKSPACE_BASE_URL="user1" \
+  --shm-size 512m \
+  --restart always \
+  mltooling/ml-workspace:0.13.2 || true
+
+docker run -d \
+ -p 8091:8080 \
+  --name "ml-workspace-user2" \
+  -v "${TOP_DIR}/files/user2:/workspace" \
+  -v "${TOP_DIR}/files/common:/workspace/common:ro" \
+  --env AUTHENTICATE_VIA_JUPYTER="" \
+  --env WORKSPACE_BASE_URL="user2" \
+  --shm-size 512m \
+  --restart always \
+  mltooling/ml-workspace:0.13.2 || true
+
+#-------------
+printf "\n \n Start the traefik gateway server\n "
+printf "...........\n "
+cd "${TOP_DIR}/servers/config/gateway" || exit
+cp "${TOP_DIR}/deploy/config/gateway/auth" auth
+cp "${TOP_DIR}/deploy/config/gateway/fileConfig.yml" "dynamic/fileConfig.yml"
+
+sudo docker run -d \
+ --name "traefik-gateway" \
+ --network=host -v "$PWD/traefik.yml:/etc/traefik/traefik.yml" \
+ -v "$PWD/auth:/etc/traefik/auth" \
+ -v "$PWD/dynamic:/etc/traefik/dynamic" \
+ -v /var/run/docker.sock:/var/run/docker.sock \
+ traefik:v2.5 || true
+
+#----------
+printf "\n \n Create crontabs to run the application in daemon mode.\n "
+printf "...........\n "
+cd "$TOP_DIR" || exit
+bash deploy/create-cronjob.sh
+
+printf "\n \n The installation is complete.\n \n \n "