diff --git a/README.md b/README.md index 6d6117ee..30bef4d2 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,10 @@ -# [GIN](https://gin.g-node.org) gin-proc Microservice +# gin-proc Microservice for [GIN](https://gin.g-node.org)
[![G-Node](./images/favicon.png)](https://gin.g-node.org) -This repository contains documentation for using the **gin-proc** microservice for **[GIN](https://gin.g-node.org)** s as well as its setup and support scripts. - -Please file an issue if you are experiencing a problem or would like to discuss something related to the microservice. - -Pull requests are encouraged if you have changes you believe would improve the setup process or increase compatibility across deployment environments. +This repository contains documentation for using the **gin-proc** microservice for **[GIN](https://gin.g-node.org)** with INCF **as a part of Google Summer of Code (GSoC) 2019 programme**, as well as its setup and support scripts.
@@ -16,13 +12,21 @@ Pull requests are encouraged if you have changes you believe would improve the s * **[Introduction](#introduction)** - [Problem statement](#problem) - [Rationale and Significance](#rationale) -* **[Installation](#install)** - - [GIN's hosted cloud](#cloud) - - [Local environment](#local) -* **[Usage](#usage)** -* **[Tests](#tests)** -* **[FAQ](#questions)** -* **[License](#project-license)** +* **[Installation](docs/install.md)** + - [GIN's hosted cloud](docs/install.md#cloud) + - [Local environment](docs/install.md#local) + - [Docker Compose](docs/install.md#docker-compose) + - [Manual](docs/install.md#manual) +* **[Usage](docs/usage.md)** + - [Workflows](docs/usage.md#workflows) + - [User's Files](docs/usage.md#files) +* **[Operations](docs/operations.md)** + - [After Login](docs/operations.md#after-login) + - [API](http://:8000/docs/api/) + - [Inside Pipeline](docs/operations.md#pipeline) +* **[FAQ](docs/faq.md)** +* **[Authors & Contribution](#authors)** +* **[License](#license)**
@@ -45,107 +49,11 @@ INCF is hosting a GIN service designed above GOGS with Git to serve as a reposit This tool/micro-service is required since, given the GIN user base of neuroscientists and other pro-fessionals from the related fields, shouldn’t be involved in writing thousands of repeated workflows for their data, and then testing it manually. This tool will increase their efficiency by almost exponential levels by eradicating redundancy from their work. -
- - - - -## Run gin-proc microservice - -Make sure your keys are installed with the GIN container. Micro-service, for now, skips ensuring/installing new keys on the GIN server (Its still in testing). - -From project's root... -```export GIN_SERVER=:``` - -```cd back-end && python server.py``` - -On a new console, go back to project's root and.. -```cd front-end && npm run dev``` - -Log in at your front-end app's SERVER IP displayed in console on endpoint `/login`. -Only log in with your GIN credentials.
- -## Usage - -In every repository that you create on gin, you have to mandatorily add a **.drone.yml** file which contains the pipelines and build jobs for drone to run. - -Use the sample `.drone.yml` file [attached](./samples/.drone.yml) in this repository. Or copy the contents from below. - -``` -kind: pipeline -name: default - -clone: - disable: true - -steps: -- name: clone - image: docker:git - environment: - REPO: test - GIN_USER: "" - GIN_SERVER: "172.19.0.2" - SSH_KEY: - from_secret: DRONE_PRIVATE_SSH_KEY - commands: - - echo "[+] Starting SSH Agent" - - eval $(ssh-agent -s) - - - echo "[+] Installing SSH Keys" - - mkdir /root/.ssh && echo "$SSH_KEY" > /root/.ssh/id_rsa && chmod 0600 /root/.ssh/id_rsa - - echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config - - ssh-keyscan -H "$GIN_SERVER" >> /root/.ssh/known_hosts - - ssh-add /root/.ssh/id_rsa - - # Uncomment the next line if you want to debug SSH Connection failures. - # - ssh -Tv git@"$GIN_SERVER" -p 22 -i /root/.ssh/id_rsa - - - git clone git@"$GIN_SERVER":/"$GIN_USER"/"$REPO".git - - echo "[+] Clone complete" - -# You can replace the following pipeline step and write your own beyond this point. - -- name: proc - image: ubuntu - environment: - REPO: test - commands: - - apt-get update - - apt-get install python -y - - which python -``` - -
- - -## Tests - -The attached sample **.drone.yml** file can be simply used to test whether things are working fine. - -We are writing more scripts to perform testing. If anyone wishes to contribute test scripts, a pull request is more than welcome. - -
- - -## FAQ - -Q: Drone throws `permission denied (publickey)` error. How do I resolve this? - -A: Add the line `- ssh -Tv git@172.19.0.2 -p 22 -i /root/.ssh/id_rsa` in your clone job in .drone.yml file before running `git clone`. It will present you with complete debug logs of the SSH connection drone tries to make from its container to your GIN container. Errors presented there can help you resolve the issue faster. - ---- - -Q: I get a `Host Key authenticated failed` error during `git clone` in drone pipeline. - -A: Either your SSH keys that you have added to GIN and to Drone as a secret don't match, or you can also check if the key pairs you installed needed your sudo password to unlock the keys. If that's the case, create new key pairs without passphrases. And install them. - -
- - -## Authors and Contributors + +## Authors and Contributions @@ -153,11 +61,15 @@ A: Either your SSH keys that you have added to GIN and to Drone as a secret don'
Achilleas KoutsouGitHub/achilleas-k
Mrinal WahalGitLab/wahalMrinalWahal.com
-Contributions are welcomed from anyone wanting to improve this project! +Contributions are welcome from anyone wanting to improve this project! + +Please file an issue if you are experiencing a problem or would like to discuss something related to the microservice. + +Pull requests are encouraged if you have changes you believe would improve the setup process or increase compatibility across deployment environments.
- + ## License This microservice is licensed under the BSD 3-Clause license. All rights not explicitly granted in the MIT license are reserved. See the included [LICENSE.md](./LICENSE.md) file for more details. @@ -166,7 +78,7 @@ This microservice is licensed under the BSD 3-Clause license. All rights not exp *Supported with :heart: by [Achilleas Koutsou](https://github.com/achilleas-k), [Michael Sonntag](https://github.com/mpsonntag) and the [G-Node](https://github.com/orgs/G-Node/people) team.* -*This project is affiliated with [G-Node](http://www.g-node.org/) and [GIN](https://gin.g-node.org).* +*This project was completed as part of *Google Summer of Code (GSoC) 2019.* and is affiliated with [G-Node](http://www.g-node.org/) and [GIN](https://gin.g-node.org).*
diff --git a/back-end/config.py b/back-end/config.py index 7263968a..00ee72a9 100644 --- a/back-end/config.py +++ b/back-end/config.py @@ -1,15 +1,58 @@ +# ------------------------------------------------------------------# +# Service: gin-proc +# Project: GIN - https://gin.g-node.org +# Documentation: https://github.com/G-Node/gin-proc/blob/master/docs +# Package: Config +# ------------------------------------------------------------------# + + import os import yaml from logger import log +from errors import ConfigurationError + + +def preparationCommands(): + + """ + Returns list of exact bash commands required initially in the + execution step to prepare workspace for pipeline. + """ + + return [ + 'eval $(ssh-agent -s)', + 'mkdir -p /root/.ssh && echo "$SSH_KEY" > \ +/root/.ssh/id_rsa && chmod 0600 /root/.ssh/id_rsa', + 'mkdir -p /etc/ssh', + 'echo "StrictHostKeyChecking no" >> \ +/etc/ssh/ssh_config', + 'ssh-add /root/.ssh/id_rsa', + 'git config --global user.name "gin-proc"', + 'git config --global user.email "gin-proc@local"', + 'ssh-keyscan -t rsa "$DRONE_GOGS_SERVER" > \ +/root/.ssh/authorized_keys', + 'if [ -d "$DRONE_REPO_NAME" ]; then cd "$DRONE_REPO_NAME"/ \ +&& git fetch --all && git checkout "$DRONE_COMMIT"; \ +else git clone "$DRONE_GIT_SSH_URL" \ +&& cd "$DRONE_REPO_NAME"/ && pip3 install -r requirements.txt; fi', + ] def createVolume(name, path): + """ + Returns a new volume dictionary. + """ + return {'name': name, 'path': path} def createEnv(name, secret): + """ + Returns a new environment variable dictionary with name and secret. + """ + return {name: {'from_secret': secret}} @@ -22,6 +65,10 @@ def createStep( commands=None ): + """ + Generates a new pipeline step configuration. + """ + PAYLOAD = {} PAYLOAD['name'] = name PAYLOAD['image'] = image @@ -37,7 +84,12 @@ def createStep( return PAYLOAD -def join_files(files, location=''): +def joindronefiles(files, location=''): + + """ + Join filenames from user's entered list in a single string + to make it processible. + """ return ' '.join( '"{}"'.format( @@ -46,23 +98,28 @@ def join_files(files, location=''): def addBackPush(files, commands): + """ + Adds commands to execution step for pushing the user's output files + back to gin-proc branch in the GIN repository. + """ + if len(files) > 0: - input_files = join_files(files) + inputdronefiles = joindronefiles(files) commands.append('TMPLOC=`mktemp -d`') - commands.append(('mv {} "$TMPLOC"').format(input_files)) + commands.append(('mv {} "$TMPLOC"').format(inputdronefiles)) commands.append('git checkout gin-proc || git checkout -b gin-proc') commands.append('git reset --hard') commands.append('mkdir "$DRONE_BUILD_NUMBER"') - input_files = join_files(files, "$TMPLOC") + inputdronefiles = joindronefiles(files, "$TMPLOC") commands.append('mv {} "$DRONE_BUILD_NUMBER"/'.format( - input_files)) + inputdronefiles)) commands.append('git annex add -c annex.largefiles="largerthan=10M" \ - "$DRONE_BUILD_NUMBER"/') +"$DRONE_BUILD_NUMBER"/') commands.append('git commit "$DRONE_BUILD_NUMBER"/ -m "Back-Push"') commands.append('git push origin gin-proc') commands.append('git annex copy --to=origin --all') @@ -72,12 +129,16 @@ def addBackPush(files, commands): def addAnnex(files, commands): + """ + Adds bash commands to get annex files required in workflow. + """ + try: if len(files) > 0: - input_files = join_files(files) + inputdronefiles = joindronefiles(files) commands.append('git annex init "$DRONE_REPO_NAME"-drone-annexe') - commands.append("git annex get {}".format(input_files)) + commands.append("git annex get {}".format(inputdronefiles)) except Exception as e: log('exception', e) @@ -88,6 +149,13 @@ def addAnnex(files, commands): def createWorkflow(workflow, commands, user_commands=None): + """ + Adds approriate bash commands as per user's specified workflow + i.e. + either Snakemake file's location + or Custom commands. + """ + try: if workflow == 'snakemake': if user_commands: @@ -108,6 +176,10 @@ def createWorkflow(workflow, commands, user_commands=None): def integrateVolumes(volumes): + """ + Returns a new volume dictionary provided name and path values. + """ + PAYLOAD = [] for volume in volumes: @@ -129,6 +201,25 @@ def generateConfig( notifications ): + """ + Automates generation of a fresh configuration for Drone + by adding necessary vanilla state pipeline steps and + integrating required volumes as per requirements. + + Two of the most important steps added in this functions + are: + + (a) restore-cache - for restoring entire cached volume + to speed up (or potentially) avoid the future repo cloning + opertaion. + + (b) rebuild-cache - for rebuilding the latest volume cache + after workflow execution has compeleted. + + Any notification steps or triggers or volumes that have to be + mounted are only added after the step which has rebuilt cache. + """ + try: log("debug", "Writing fresh configuration.") @@ -159,23 +250,7 @@ def generateConfig( 'SSH_KEY', 'DRONE_PRIVATE_SSH_KEY' ), - commands=[ - 'eval $(ssh-agent -s)', - 'mkdir -p /root/.ssh && echo "$SSH_KEY" > \ -/root/.ssh/id_rsa && chmod 0600 /root/.ssh/id_rsa', - 'mkdir -p /etc/ssh', - 'echo "StrictHostKeyChecking no" >> \ -/etc/ssh/ssh_config', - 'ssh-add /root/.ssh/id_rsa', - 'git config --global user.name "gin-proc"', - 'git config --global user.email "gin-proc@local"', - 'ssh-keyscan -t rsa "$DRONE_GOGS_SERVER" > \ -/root/.ssh/authorized_keys', - 'if [ -d "$DRONE_REPO_NAME" ]; then cd "$DRONE_REPO_NAME"/ \ -&& git fetch --all && git checkout "$DRONE_COMMIT"; \ -else git clone "$DRONE_GIT_SSH_URL" \ -&& cd "$DRONE_REPO_NAME"/ && pip3 install -r requirements.txt; fi', - ] + commands=preparationCommands() ), createStep( name='rebuild-cache', @@ -189,7 +264,6 @@ def generateConfig( ], 'volumes': integrateVolumes([ ('cache', '/gin-proc/cache'), - ('repo', '/gin-proc/repo') ]), 'trigger': { 'branch': ['master'], @@ -228,6 +302,11 @@ def modifyConfigFiles( commands ): + """ + Modifies the workflow and notification steps as required + on existing pipeline configuration. + """ + try: log("debug", "Adding user's files.") @@ -245,6 +324,12 @@ def modifyConfigFiles( def addNotifications(notifications, data): + """ + Adds additional pipeline step for notifying the user + post completion of build job on service of choice + - mostly Slack. + """ + notifications = [n for n in notifications if n['value']] for step in data: @@ -278,49 +363,68 @@ def ensureConfig( notifications=[] ): - try: + """ + First line of defense! + + Runs following checks: - __file = os.path.join(config_path, '.drone.yml') - if not os.path.exists(__file) or os.path.getsize(__file) <= 0: - log("warning", "CI Config either not found in repo or is corrupt.") - - with open(os.path.join(config_path, '.drone.yml'), 'w') \ - as new_config: - __generated_config = generateConfig( - workflow=workflow, - commands=userInputs, - annexFiles=annexFiles, - backPushFiles=backPushFiles, - notifications=notifications - ) - - if not __generated_config: - return False - else: - yaml.dump( - __generated_config, - new_config, - default_flow_style=False) - - return True + 1. Whether or not a pipeline configuration already exists. + 2. If it exists, is it corrupt or un-processible? + 3. If not, do the preparation commands required in + execution step match our standards. + + Resolutions to above checks: + + For case 1: Initiates generation of a fresh configuration, if doesn't. + For cases 2 and 3: Raises error and initiates overriting of existing + configuration with a yet fresh one -- this will delete user's + manual changes to configuration. + + Complete documentation for all operations in this function + can also be accessed at: + + https://github.com/G-Node/gin-proc/blob/master/docs/operations.md + + """ + + dronefile = os.path.join(config_path, '.drone.yml') + execution_step = None + + try: + if not os.path.exists(dronefile) or os.path.getsize(dronefile) <= 0: + raise ConfigurationError("CI Config either not found in repo or is corrupt.") else: - __file = os.path.join(config_path, '.drone.yml') log("debug", "Updating already existing CI Configuration.") config = [] - with open(__file, 'r') as stream: + with open(dronefile, 'r') as stream: config = yaml.load(stream, Loader=yaml.FullLoader) - with open(__file, 'w') as stream: + execution_step = [ + step for step in config['steps'] if + step['name'] == 'execute' + ][0] + + if execution_step['commands'][:len( + preparationCommands())] != preparationCommands(): + + raise ConfigurationError( + "Existing CI Config does not match correct \ +preparation mechanism for pipeline.") + + with open(dronefile, 'w') as stream: - config['steps'][1]['commands'] = modifyConfigFiles( + config['steps'][config['steps'].index( + execution_step)]['commands'] = modifyConfigFiles( workflow=workflow, annexFiles=annexFiles, backPushFiles=backPushFiles, commands=userInputs, - data=config['steps'][1]['commands'][:9] + data=config['steps'][config['steps'].index(execution_step)] + ['commands'][:len( + preparationCommands())] ) config['steps'] = addNotifications( @@ -333,7 +437,24 @@ def ensureConfig( stream, default_flow_style=False) - return True + except ConfigurationError as e: + log('error', e) + log('info', 'Generating fresh configuration.') - except Exception as e: - log('exception', e) \ No newline at end of file + with open(os.path.join(config_path, '.drone.yml'), 'w') \ + as new_config: + generated_config = generateConfig( + workflow=workflow, + commands=userInputs, + annexFiles=annexFiles, + backPushFiles=backPushFiles, + notifications=notifications + ) + + if not generated_config: + return False + else: + yaml.dump( + generated_config, + new_config, + default_flow_style=False) diff --git a/back-end/errors.py b/back-end/errors.py new file mode 100644 index 00000000..ac58b06a --- /dev/null +++ b/back-end/errors.py @@ -0,0 +1,35 @@ +# ------------------------------------------------------------------# +# Service: gin-proc +# Project: GIN - https://gin.g-node.org +# Documentation: https://github.com/G-Node/gin-proc/blob/master/docs +# Package: Errors and Exceptions +# ------------------------------------------------------------------# + + +class ServiceError(Exception): + + def __init__(self, message, payload=None): + self.message = message + self.payload = payload + + def __str__(self): + return str(self.message) + + +class ConfigurationError(Exception): + + def __init__(self, message): + self.message = message + + def __str__(self): + return str(self.message) + + +class ServerError(Exception): + + def __init__(self, message, status=None): + self.message = message + self.status = status + + def __str__(self): + return str(self.message) diff --git a/back-end/logger.py b/back-end/logger.py index eefa62a0..9795f3a4 100644 --- a/back-end/logger.py +++ b/back-end/logger.py @@ -1,3 +1,11 @@ +# ------------------------------------------------------------------# +# Service: gin-proc +# Project: GIN - https://gin.g-node.org +# Documentation: https://github.com/G-Node/gin-proc/blob/master/docs +# Package: Logger +# ------------------------------------------------------------------# + + import logging import os diff --git a/back-end/server.py b/back-end/server.py index cb98e4f9..5553158b 100644 --- a/back-end/server.py +++ b/back-end/server.py @@ -1,12 +1,33 @@ +# ------------------------------------------------------------------# +# Service: gin-proc +# Project: GIN - https://gin.g-node.org +# Documentation: https://github.com/G-Node/gin-proc/blob/master/docs +# API Documentation: /docs/api +# Package: Server (Flask API) +# ------------------------------------------------------------------# + + from service import configure, ensureToken, ensureKeys, getRepos, userData from service import log, ensureSecrets -from flask import Flask, request, abort, jsonify +from flask import Flask, request, abort, jsonify, Blueprint +from http import HTTPStatus + +import errors + +from flask_docs import ApiDoc from flask_cors import CORS, cross_origin app = Flask(__name__) cors = CORS(app) app.config['CORS_HEADERS'] = 'Content-Type' +app.config['API_DOC_MEMBER'] = ['api', 'platform', 'auth'] + +ApiDoc(app) + +api = Blueprint('api', __name__) +auth = Blueprint('auth', __name__) +platform = Blueprint('platform', __name__) class User(object): @@ -19,27 +40,35 @@ def login(self): self.username = request.json['username'] password = request.json['password'] - self.GIN_TOKEN = ensureToken(user.username, password) - log("debug", 'GIN token ensured.') + try: + self.GIN_TOKEN = ensureToken(user.username, password) + log("debug", 'GIN token ensured.') - if ensureKeys(self.GIN_TOKEN) and ensureSecrets(self.username): - return ({'token': self.GIN_TOKEN}, 200) + except errors.ServerError as e: + log('critical', e) + return (e, HTTPStatus.INTERNAL_SERVER_ERROR) + + if ( + ensureKeys(self.GIN_TOKEN) and ensureSecrets(self.username) + ): + return ({'token': self.GIN_TOKEN}, HTTPStatus.OK) else: - return ('login failed', 400) + return ('login failed', HTTPStatus.UNAUTHORIZED) def logout(self): user.username = None user.GIN_TOKEN = None user.DRONE_TOKEN = None - return ("logged out", 200) + return ("logged out", HTTPStatus.OK) def details(self): - return (userData(self.GIN_TOKEN), 200) + return (userData(self.GIN_TOKEN), HTTPStatus.OK) def run(self, request): - if configure( + try: + configure( repoName=request.json['repo'], notifications=request.json['notifications'], commitMessage=request.json['commitMessage'], @@ -52,11 +81,12 @@ def run(self, request): request.json['backpushFiles'].values()))), token=self.GIN_TOKEN, username=self.username - ): + ) return ("Success: workflow pushed to {}".format( - request.json['repo']), 200) - else: - return ("Workflow failed.", 400) + request.json['repo']), HTTPStatus.OK) + + except errors.ServiceError as e: + return (e, HTTPStatus.INTERNAL_SERVER_ERROR) def repos(self): @@ -71,43 +101,110 @@ def repos(self): user = User() -@app.route('/logout', methods=['POST']) -def logMeOut(): +@auth.route('/logout', methods=['POST']) +def logout(): + + """ + Clears user's credentials including auth token for the session. + """ + if request.method == "POST": return user.logout() -@app.route('/login', methods=['POST']) +@auth.route('/login', methods=['POST']) @cross_origin() -def logMeIn(): +def login(): + + """ + Authenticates user with their GIN credentials. + + @@@ + Ensures following checks in chronological order are passed for succesfull + authentication: + + - #### Ensure Token + Runs a check whether GIN already has a `personal access token (PAT)` + installed on your account specifically for `gin-proc`. + In case it doesnt (which is highly likely if you are logging in to + `gin-proc` for the first time), it shall automatically create + and install a fresh token for you. + + - #### Ensure SSH Keys + Ensures whether a specific SSH key pair is already installed for use + by `gin-proc` in GIN. In case it doesn't (which is highly likely if + you are logging in to `gin-proc` for the first time), + it shall create a fresh key pair for you and install the appropriate + key `public key` in GIN so that gin-proc has read/write access to your GIN repos. + + - #### Ensure Drone Secrets + Runs a check to ensure that all of your Drone repositories are + activated and that they have your subsequent `private-key` installed + in them as a **secret**. This secret is used by Drone for cloning and + pushing operations on your GIN repos whilst its running your + build jobs inside its runners. + @@@ + """ + if request.method == "POST": - return user.login() - else: - abort(500) + try: + return user.login() + except errors.ServerError as e: + abort(e.status) + + +@auth.route('/user', methods=['GET']) +def Get_User(): + """ + Returns logged-in user's data from GIN. + """ -@app.route('/user', methods=['GET']) -def getUser(): if request.method == "GET": - return user.details() - else: - abort(500) + try: + return user.details() + except errors.ServerError as e: + abort(e.status) + + + +@api.route('/execute', methods=['POST']) +@cross_origin() +def Execute_Workflow(): + """ + Runs the workflow post user's submission from front-end UI. -@app.route('/execute', methods=['POST']) -def execute(): + @@@ + For complete documentation of execution steps, read + [operations doc](https://github.com/G-Node/gin-proc/blob/master/docs/operations.md). + @@@ + """ if request.method == "POST": - return user.run(request) - else: - abort(500) + + try: + return user.run(request) + except errors.ServerError as e: + abort(e.status) -@app.route('/repos', methods=['GET']) +@api.route('/repos', methods=['GET']) @cross_origin() -def repos(): +def repositories(): + + """ + Returns list of user's repositories from GIN. + """ + if request.method == "GET": return user.repos() + +app.register_blueprint(api, url_prefix='/api') +app.register_blueprint(auth, url_prefix='/auth') +app.register_blueprint(platform, url_prefix='/platform') + + if __name__ == '__main__': - app.run(debug=True, port=8000, host="0.0.0.0") \ No newline at end of file + app.run(debug=False, port=8000, host="0.0.0.0") \ No newline at end of file diff --git a/back-end/service.py b/back-end/service.py index b326374b..8d4f2c46 100644 --- a/back-end/service.py +++ b/back-end/service.py @@ -1,9 +1,14 @@ -# -------------------------------# +# ------------------------------------------------------------------# +# Service: gin-proc +# Project: GIN - https://gin.g-node.org +# Documentation: https://github.com/G-Node/gin-proc/blob/master/docs +# Package: Service +# ------------------------------------------------------------------# # Env variables assigned # export GIN_SERVER=http://172.19.0.2:3000 # export DRONE_SERVER=http://172.19.0.3 # DRONE_TOKEN=AAAAAAAAAA000000000000000XXXXXXXXX -# -------------------------------# +# ------------------------------------------------------------------# import requests @@ -11,9 +16,11 @@ from shutil import rmtree import tempfile +from http import HTTPStatus from config import ensureConfig from logger import log from subprocess import call +from errors import ServiceError, ServerError from cryptography.hazmat.primitives import serialization from cryptography.hazmat.primitives.asymmetric import rsa @@ -29,6 +36,10 @@ def userData(token): + """ + Returns logged-in user's data from GIN. + """ + return requests.get( GIN_ADDR + "/api/v1/user", headers={'Authorization': 'token {}'.format(token)} @@ -37,49 +48,76 @@ def userData(token): def ensureToken(username, password): - res = requests.get( - GIN_ADDR + "/api/v1/users/{}/tokens".format(username), - auth=(username, password)).json() + """ + Retrieves the personal access token `gin-proc` + from user's GIN account to be used further in session. + + In case, the specific token for gin-proc doesn't exists, + it registers a fresh token to GIN for that user. + """ - for token in res: - if token['name'] == 'gin-proc': - return token['sha1'] + try: + res = requests.get( + GIN_ADDR + "/api/v1/users/{}/tokens".format(username), + auth=(username, password)).json() + + for token in res: + if token['name'] == 'gin-proc': + return token['sha1'] + + res = requests.post( + GIN_ADDR + "/api/v1/users/{}/tokens".format(username), + auth=(username, password), + data={'name': 'gin-proc'} + ).json() - res = requests.post( - GIN_ADDR + "/api/v1/users/{}/tokens".format(username), - auth=(username, password), - data={'name': 'gin-proc'} - ).json() + return res['sha1'] - return res['sha1'] + except Exception.ConnectionError as e: + raise ServerError(e) def writeSecret(key, repo, user): - res = requests.post( - DRONE_ADDR + "/api/repos/{0}/{1}/secrets".format(user, repo), - headers={ - 'Authorization': 'Bearer {}'.format(os.environ['DRONE_TOKEN']), - 'Content-Type': "application/json" - }, - json={ - "name": "DRONE_PRIVATE_SSH_KEY", - "data": key, - "pull_request": False - } - ) + """ + Writes the key as a secret title `DRONE_PRIVATE_SSH_KEY` + to specified repository in Drone. + """ - if res.status_code == 200: + try: + res = requests.post( + DRONE_ADDR + "/api/repos/{0}/{1}/secrets".format(user, repo), + headers={ + 'Authorization': 'Bearer {}'.format(os.environ['DRONE_TOKEN']), + 'Content-Type': "application/json" + }, + json={ + "name": "DRONE_PRIVATE_SSH_KEY", + "data": key, + "pull_request": False + } + ) + + except Exception.ConnectionError as e: + raise ServerError(e) + + if res.status_code == HTTPStatus.OK: log('debug', 'Secret installed in `{}`'.format(repo)) return True else: - log('warning', 'Secret could not be installed in `{}`'.format(repo)) log('critical', res.json()['message']) - return False + raise ServerError('Secret could not be installed in `{}`'.format(repo), + HTTPStatus.INTERNAL_SERVER_ERROR) def updateSecret(secret, data, user, repo): + """ + Ensure the secret DRONE_PRIVATE_SSH_KEY already exists, + and if true, update the secret with latest key. + Else, register the key as a secret. + """ + res = requests.patch( DRONE_ADDR + "/api/repos/{user}/{repo}/secrets/{secret}".format( user=user, @@ -91,54 +129,66 @@ def updateSecret(secret, data, user, repo): "pull_request": False }) - if res.status_code == 200: + if res.status_code == HTTPStatus.OK: log('debug', 'Secret updated in `{}`'.format(repo)) return True else: - log('warning', 'Secret could not be updated in `{}`'.format(repo)) - log('critical', 'Execution may not work properly from here.') - return False + raise ServerError('Secret could not be updated in `{}`'.format(repo), + HTTPStatus.INTERNAL_SERVER_ERROR) def ensureSecrets(user): + """ + Runs a check on all of user's ACTIVATED Drone repositories + if each of them has the secret DRONE_PRIVATE_SSH_KEY and is updated + with the latest key. + + Initiates the installation process for secret, if it doesn't exists. + """ + repos = requests.get( DRONE_ADDR + "/api/user/repos", headers={ 'Authorization': 'Bearer {}'.format(os.environ['DRONE_TOKEN']) }).json() - for repo in repos: - if repo['active']: + for repo in [repo for repo in repos if repo['active']]: + + secrets = requests.get( + DRONE_ADDR + "/api/repos/{0}/{1}/secrets".format( + user, repo['name']), + headers={'Authorization': 'Bearer {}'.format( + os.environ['DRONE_TOKEN'])} + ).json() - secrets = requests.get( - DRONE_ADDR + "/api/repos/{0}/{1}/secrets".format(user, repo['name']), - headers={'Authorization': 'Bearer {}'.format( - os.environ['DRONE_TOKEN'])} - ).json() + with open(os.path.join(SSH_PATH, PRIV_KEY), 'r') as key: - with open(os.path.join(SSH_PATH, PRIV_KEY), 'r') as key: + for secret in secrets: + if secret['name'] == 'DRONE_PRIVATE_SSH_KEY': + log('debug', 'Secret found in repo `{}`'.format( + repo['name'])) - for secret in secrets: - if secret['name'] == 'DRONE_PRIVATE_SSH_KEY': - log('debug', 'Secret found in repo `{}`'.format( - repo['name'])) + updateSecret( + secret=secret['name'], + data=key.read(), + repo=repo['name'], + user=user + ) + break - return updateSecret( - secret=secret['name'], - data=key.read(), - repo=repo['name'], - user=user - ) - else: - return(writeSecret(key.read(), repo['name'], user)) + log('debug', 'Secret not found in `{}`'.format(repo['name'])) + writeSecret(key.read(), repo['name'], user) - log('debug', 'Secret not found in `{}`'.format(repo['name'])) - return(writeSecret(key.read(), repo['name'], user)) + return True def getKeysFromServer(token): + """ + Fetches all SSH public keys from user's GIN account. + """ + return requests.get( GIN_ADDR + "/api/v1/user/keys", headers={'Authorization': 'token {}'.format(token)} @@ -147,6 +197,11 @@ def getKeysFromServer(token): def ensureKeysOnServer(token): + """ + Confirms whether the public key 'gin-proc' is installed + on usre's GIN account or not. + """ + for key in getKeysFromServer(token): if key['title'] == PRIV_KEY: return True @@ -154,6 +209,10 @@ def ensureKeysOnServer(token): def deleteKeysOnServer(token): + """ + Deletes key 'gin-proc' from user's GIN account. + """ + for key in getKeysFromServer(token): if key['title'] == PRIV_KEY: response = requests.delete( @@ -163,75 +222,100 @@ def deleteKeysOnServer(token): if response.status_code == 204: log('warning', 'Deleted keys from server.') + return True else: log('error', response.text) - log('critical', - "You'll have to manually delete the keys from the server.") - - return + raise ServerError( + "You'll have to manually delete the keys from the server.", + HTTPStatus.SERVICE_UNAVAILABLE) def ensureKeysOnLocal(path): + """ + Confirms whether SSH Private key exists locally or not. + """ + return os.path.exists(os.path.join(path, PRIV_KEY)) def installFreshKeys(SSH_PATH, token): - key = rsa.generate_private_key( - backend=default_backend(), - public_exponent=65537, - key_size=2048 - ) - private_key = key.private_bytes( - serialization.Encoding.PEM, - serialization.PrivateFormat.PKCS8, - serialization.NoEncryption() - ) - public_key = key.public_key().public_bytes( - serialization.Encoding.OpenSSH, - serialization.PublicFormat.OpenSSH + """ + Generates a fresh pair of public and private keys. + And installs them on user's GIN account. + """ + + key = rsa.generate_private_key( + backend=default_backend(), + public_exponent=65537, + key_size=2048 + ) + private_key = key.private_bytes( + serialization.Encoding.PEM, + serialization.PrivateFormat.PKCS8, + serialization.NoEncryption() ) + public_key = key.public_key().public_bytes( + serialization.Encoding.OpenSSH, + serialization.PublicFormat.OpenSSH + ) - os.makedirs(SSH_PATH, exist_ok=True) + os.makedirs(SSH_PATH, exist_ok=True) - with open( - os.path.join(SSH_PATH, PRIV_KEY), - 'w+') as private_key_file: + with open( + os.path.join(SSH_PATH, PRIV_KEY), + 'w+') as private_key_file: - private_key_file.write(private_key.decode('utf-8')) + private_key_file.write(private_key.decode('utf-8')) - with open( - os.path.join(SSH_PATH, PUB_KEY), - 'w+') as public_key_file: + with open( + os.path.join(SSH_PATH, PUB_KEY), + 'w+') as public_key_file: - public_key_file.write(public_key.decode('utf-8')) + public_key_file.write(public_key.decode('utf-8')) - os.chmod(SSH_PATH, 0o700) - os.chmod(os.path.join(SSH_PATH, PRIV_KEY), 0o600) - os.chmod(os.path.join(SSH_PATH, PUB_KEY), 0o600) + os.chmod(SSH_PATH, 0o700) + os.chmod(os.path.join(SSH_PATH, PRIV_KEY), 0o600) + os.chmod(os.path.join(SSH_PATH, PUB_KEY), 0o600) - requests.post( - GIN_ADDR + "/api/v1/user/keys", - headers={'Authorization': 'token ' + token}, - data={'title': PRIV_KEY, 'key': public_key} - ) + requests.post( + GIN_ADDR + "/api/v1/user/keys", + headers={'Authorization': 'token ' + token}, + data={'title': PRIV_KEY, 'key': public_key} + ) - log('info', 'Fresh key pair installed with pub key {}'.format(PUB_KEY)) - return PUB_KEY + log('info', 'Fresh key pair installed with pub key {}'.format(PUB_KEY)) def ensureKeys(token): + """ + Runs following checks for required SSH key pair: + + 1. Do keys exist both locally and GIN server? + 2. Do keys exist only locally but not on server? + 3. Do keys exist only on server but not locally? + + Resolution for above cases: + + Case 1: Returns positive response. + Case 2: Delete keys locally and install a fresh pair both + locally and on server. + Case 3: Delete keys on server and install a fresh pair both + locally and on server. + """ + try: if ensureKeysOnServer(token) and ensureKeysOnLocal(SSH_PATH): log("debug", "Keys ensured both on server and locally.") + return True + elif ensureKeysOnServer(token) and not ensureKeysOnLocal(SSH_PATH): log("debug", "Key is installed on the server but not locally.") deleteKeysOnServer(token) - installFreshKeys(SSH_PATH, token) elif not ensureKeysOnServer(token) and ensureKeysOnLocal(SSH_PATH): log("debug", "Key is installed locally but not on the server.") @@ -240,20 +324,19 @@ def ensureKeys(token): os.remove(os.path.join(SSH_PATH, PUB_KEY)) log("warning", "Removed local keys.") - installFreshKeys(SSH_PATH, token) - - else: - installFreshKeys(SSH_PATH, token) + installFreshKeys(SSH_PATH, token) - return True - - except Exception as e: - log('exception', e) - return False + except: + log('critical', 'Failed to ensure keys.') + raise ServerError('Cannot ensure keys.', HTTPStatus.INTERNAL_SERVER_ERROR) def getRepos(user, token): + """ + Fetches list of all repositories from user's GIN account. + """ + return requests.get( GIN_ADDR + "/api/v1/users/{}/repos".format(user), headers={'Authorization': 'token {}'.format(token)}, @@ -262,6 +345,10 @@ def getRepos(user, token): def getRepoData(user, repo, token): + """ + Fetches complete data of a repository from user's GIN account. + """ + return requests.get( GIN_ADDR + "/api/v1/repos/{0}/{1}".format(user, repo), headers={'Authorization': 'token {}'.format(token)} @@ -269,6 +356,11 @@ def getRepoData(user, repo, token): def clone(repo, author, path): + + """ + Clones the repository in question in a temporary location (path). + """ + clone_path = os.path.join(path, author, repo['name']) os.makedirs(clone_path, exist_ok=True) @@ -280,29 +372,26 @@ def clone(repo, author, path): def push(path, commitMessage): - try: - call(['git', 'add', '.'], cwd=path) - call(['git', 'commit', '-m', commitMessage], cwd=path) - call(['git', 'push'], cwd=path) + """ + Commits and pushes the updates from temporary location (path) the + repository is stored at on to the GIN server. + """ - log("info", "Updates pushed from {}".format(path)) - return True + call(['git', 'add', '.'], cwd=path) + call(['git', 'commit', '-m', commitMessage], cwd=path) + call(['git', 'push'], cwd=path) - except Exception as e: - log('critical', e) - return False + log("info", "Updates pushed from {}".format(path)) def clean(path): - try: - rmtree(path) - log("debug", "Repo cleaned from {}".format(path)) - return True + """ + Removes the cloned repository data and free the temporary space (path). + """ - except Exception as e: - log('critical', e) - return False + rmtree(path) + log("debug", "Repo cleaned from {}".format(path)) def configure( @@ -317,15 +406,46 @@ def configure( workflow ): - repo = getRepoData(username, repoName, token) + """ + First line of action! + + This function is reposible for integrating the entire workflow + together and executing the following in chronological order: + + 1. Fetches the repository data for repo in question. + + 2. Specifies the 'GIT_SSH_COMMAND' to locally stored + private key, in order to use it for all git operations henceforth. + + 3. Generates a temporary location for all future git operations + to take place in. + + 4. Runs operation to ensure configuration. + Complete documentation for all operations executed in 'ensureConfig' + function can also be accessed at: + + https://github.com/G-Node/gin-proc/blob/master/docs/operations.md + + 5. Commits and pushes any updates in the cloned repository + after workflow configuration is complete and successfull. + + 6. Deletes the cloned repository data and deletes temporary location. + """ + + try: + repo = getRepoData(username, repoName, token) + + except Exception as e: + log('error', e) + raise ServiceError(e) os.environ['GIT_SSH_COMMAND'] = "ssh -i {}".format( - os.path.join(SSH_PATH, PRIV_KEY)) + os.path.join(SSH_PATH, PRIV_KEY)) with tempfile.TemporaryDirectory() as temp_clone_path: clone_path = clone(repo, username, temp_clone_path) - status = ensureConfig( + ensureConfig( config_path=clone_path, workflow=workflow, userInputs=userInputs, @@ -336,5 +456,3 @@ def configure( push(clone_path, commitMessage) clean(clone_path) - - return status \ No newline at end of file diff --git a/docs.md b/docs.md deleted file mode 100644 index da65ec18..00000000 --- a/docs.md +++ /dev/null @@ -1,214 +0,0 @@ -### Documentation - -[Initial work similar to README] - -## Table of Contents -* **[Introduction]()** - - [Problem statement](#problem) - - [Rationale and Significance](#rationale) -* **[Installation](#install)** - - [GIN's hosted cloud](#cloud) - - [Local environment](#local) - - [Docker Compose](#docker-compose) - - [Manual](#manual) -* **[Usage](#usage)** - - [Workflows](#workflows) - - [User's Files](#files) -* **[Operations](#operations)** - - [After Login](#after-login) - - [API](#api) - - [Inside Pipeline](#pipeline) -* **[FAQ](#questions)** -* **[License](#project-license)** - -
- - -## Installation - - -### GIN's Hosted Cloud - -We are currently deploying the utility for our cloud environment. Please check back after some time. - - -### Local Environment (D-I-Y) - -
- - -#### Docker Compose - -You can simply run `the docker-compose` file included in the repositoy which is configured to run all the 3 services := GIN, Drone and gin-proc for you. - - -#### Manual Set-Up - -Use the following tutorial to st up all 3 containers separately. - -**Prerequisites** - -* [Docker](https://docs.docker.com/) -* [g-node/gin-web](https://hub.docker.com/r/gnode/gin-web) node running in a docker container. If you don't have this then folow steps until [#install-gin](#install-gin) - -It's advisable to use all the **gin micro-services** inside docker containers and further connect all containers to interact with each other for increased fault tolerance. - -
- -**Steps** - - -**[1]** In order to allow all gin containers to interact with each other, create a **docker network** and connect all containers to this network. - -`docker network create ` - -**[2]** If you already have a **gin** service container running, then attach it to the new network. - -`docker network connect ` - - -**Or** if you do not have a **gin** service container running already, then start a new one with the following command. -To keep things easier, we'll attach a static IP `172.19.0.2` to this container so that we don't have to inspect the docker network for changes in IP later. - -`docker run --name=gin --net --ip 172.19.0.2 -p 10022:22 -p 3000:3000 -v /var/gogs:/data gnode/gin-web:rebased` - -If you already had a gin container running, then run the following command to check and copy IP address of **gin** service container. - -`docker network inspect ` - -Copy the IP of your GIN container from `containers` section. - -**[3]** Set-up GIN Service. - -If you have already set-up a gin container previuosly, then create a custom configuration file ([check GOGS docs for custom config](https://gogs.io/docs/features/custom_template)) with the following details. - - -``` -DOMAIN = 172.19.0.2 -HTTP_PORT = 3000 -ROOT_URL = http://172.19.0.2:3000/ -``` - -Keep everything else the same. - -If you have set-up the gin service for the first time through this tutorial, then access `172.19.0.2:3000` in your browser. You will be treated with the configuration page. - -Mention the same config details as [above](#config) and leave everything else the same. - -Save the details. Register for a new account and login. - -**[4]** Create your first repository on the gin service. Clone the repository anywhere on your machine. - -**[5]** Create **drone** CI/CD service container. - -``` -docker run --volume=/var/run/docker.sock:/var/run/docker.sock --volume=/gin-proc/cache:/cache --volume=/gin-proc/ssh:/ssh --env=DRONE_GIT_ALWAYS_AUTH=false --env=DRONE_GOGS_SERVER=http://172.19.0.2:3000 --env=DRONE_RUNNER_CAPACITY=2 --env=DRONE_RUNNER_NETWORKS=gin --env=DRONE_SERVER_HOST=172.19.0.3 --env=DRONE_SERVER_PROTO=http --env=DRONE_TLS_AUTOCERT=false --publish=80:80 --publish=443:443 --publish=2224:22 --restart=always --detach=true --net gin --name=drone drone/drone:latest - ``` - -Access drone at `172.19.0.3:80` in your browser. - -**[6]** Create **gin-proc** service. - -``` -docker run --net gin --name=gin-proc gnode/proc: -``` - -**Additional Environment Variables** - -`DEBUG=True` - Serves log messages from the lowest DEBUG level i.e. DEBUG, INFO, WARNING, ERROR, EXCEPTION. - -`DEBUG=False` or simply, not assigning the var - Servers log messages from and above INFO level i.e. INFO, WARNING, ERROR, EXCEPTION. - -`LOG_DIR=/path/to/your/log/dir` - Specific location to store your logs in. Otherwise, the logs are simply printed on your console in real time. - -
- - -## Usage - -Go to your `gin-proc` address on port 3000 to access the service UI. Only use your GIN credentials to login. Users are not required to signup separately for `gin-proc`. - - -### Types of Workflows - -**Snakemake** - If you choose this workflow, you can enter the location of your Snakefile in your repo. If nothing is mentioned in location on during submission, then default location will be assumed to be the root of the repository. - -**Custom** - Users can enter the exact commands they want to run inside containers post cloning of the repository. Commands will be executed in the same exact order they are added to the workflow in. - - -### User's files - -**Annex Files** - Users can mention whether any files have to be especially `git-annex`ed before executing the workflow. - -**BackPush Files** - Users can mention the output files produced post execution of the workflow to be pushed back to user's GIN repository. Files will be pushed to a separate `gin-proc` branch in the repository. - -
- - -## Operations - - -#### What happens just after you Log-In - -`gin-proc` does following operations for you in the background, in chronological order. - -**+** When you first log into the front-end, the microservice automatically logs into your GIN account whith your specified credentials. - -**+** It then runs a check whether GIN already has a `personal access token (PAT)` installed on your account specifically for `gin-proc`. In case it doesnt (which is highly likely if you are logging in to `gin-proc` for the first time), it shall automatically create and install a fresh token for you. - -**+** Immediately after that it ensures whether a specific SSH key pair is already installed for use by `gin-proc` in GIN. In case it doesn't (which is highly likely if you are logging in to `gin-proc` for the first time), it shall create a fresh key pair for you and install the appropriate key `public key` in GIN so that gin-proc has read/write access to your GIN repos. - -**+** Then `gin-proc` runs a check to ensure that all of your Drone repositories are activated and that they have your subsequent `private-key` installed in them as a **secret**. This secret is used by Drone for cloning and pushing operations on your GIN repos whilst its running your build jobs inside its runners. - -In case your repositories aren't activated in Drone, `gin-proc` activates them and installs the required secret(s) by itself. - -
- - -#### What happens after you run the workflow from front-end - -Your input data from the front-end is sent to the REST API running on port `8000` on the same address. And the API takes over the job from there. - - -### API - -[Will be hyperlinked to Flask's Auto-Generated Documentations] - -
- - -### Inside the Pipeline - -The Drone config `gin-proc` service writes for you essentially does the following inside Drone containers (/runners) in chronological order: - -**+** Run a check whether you have run any previous builds on the repository in question, and if yes - then restore the repo from cache in order to speed up the clone process afterwards. - -**+** Starts the SSH agent. - -**+** Creates an SSH direcotory and writes your `priv-key` from Drone secrets of that repository to that directory, for Drone's use to clone your repository from GIN afterwards. - -**+** Disables `Strict Host Checking` for all authorized keys. - -**+** Adds your newly written `priv-key` to the SSH agent. - -**+** Runs a keyscan on your GIN container and writes the returned SSH fingerprint to `authorized_keys` - -**+** Clones your repository from GIN. - -**+** Installs Python dependencies if a `requirements.txt` file exists in your repo. - -**+** Annex your files if you had mentioned any files to be annexed - after initialising an annex repo in the working directory. - -**+** Attaches the commands or runs the Snakefile depending on your workflow chosen. - -**+** Add the backpush files, if you added any, and push them back to your repository in a separate `gin-proc` branch after completion of build job. - -**+** Pushes the produced output files back finally. - -**+** Rebuilds the latest cache of the repo after final operations. - -**+** Integrates the required volumes := `/cache` and `/repo` - -**+** Tells Drone to only run the build job in the event if you **pushed** to the **master** branch of your repository. Not in any otherwise case. - - **+** Attaches Slack or Email notification plugin if you had selected it from the front-end. \ No newline at end of file diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..69523c79 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,23 @@ +## Documentation + +
+ +* **[Introduction](https://github.com/G-Node/gin-proc/blob/master/README.md)** + - [Service Diagram](gin-proc_workflow_draft.pdf) + - [Problem statement](https://github.com/G-Node/gin-proc/blob/master/README.md#problem) + - [Rationale and Significance](https://github.com/G-Node/gin-proc/blob/master/README.md#rationale) +* **[Installation](install.md)** + - [GIN's hosted cloud](install.md#cloud) + - [Local environment](install.md#local) + - [Docker Compose](install.md#docker-compose) + - [Manual](install.md#manual) +* **[Usage](usage.md)** + - [Workflows](usage.md#workflows) + - [User's Files](usage.md#files) +* **[Operations](operations.md)** + - [After Login](operations.md#after-login) + - [API](http://:8000/docs/api/) + - [Inside Pipeline](operations.md#pipeline) +* **[FAQ](faq.md)** +* **[Authors & Contribution](https://github.com/G-Node/gin-proc/blob/master/README.md#authors)** +* **[License](https://github.com/G-Node/gin-proc/blob/master/README.md#license)** \ No newline at end of file diff --git a/docs/faq.md b/docs/faq.md new file mode 100644 index 00000000..dcd1088b --- /dev/null +++ b/docs/faq.md @@ -0,0 +1,11 @@ +## Frequently Asked Questions + +Q: Drone throws `permission denied (publickey)` error. How do I resolve this? + +A: Add the line `- ssh -Tv git@172.19.0.2 -p 22 -i /root/.ssh/id_rsa` in your clone job in .drone.yml file before running `git clone`. It will present you with complete debug logs of the SSH connection drone tries to make from its container to your GIN container. Errors presented there can help you resolve the issue faster. + +--- + +Q: I get a `Host Key authenticated failed` error during `git clone` in drone pipeline. + +A: Either your SSH keys that you have added to GIN and to Drone as a secret don't match, or you can also check if the key pairs you installed needed your sudo password to unlock the keys. If that's the case, create new key pairs without passphrases. And install them. \ No newline at end of file diff --git a/doc/gin-proc_workflow_draft.pdf b/docs/gin-proc_workflow_draft.pdf similarity index 100% rename from doc/gin-proc_workflow_draft.pdf rename to docs/gin-proc_workflow_draft.pdf diff --git a/docs/install.md b/docs/install.md new file mode 100644 index 00000000..c254e32e --- /dev/null +++ b/docs/install.md @@ -0,0 +1,97 @@ + +## Installation + + +### GIN's Hosted Cloud + +We are currently deploying the utility for our cloud environment. Please check back after some time. + + +### Local Environment (D-I-Y) + +
+ + +#### Docker Compose + +You can simply run `the docker-compose` file included in the repositoy which is configured to run all the 3 services := GIN, Drone and gin-proc for you. + + +#### Manual Set-Up + +Use the following tutorial to st up all 3 containers separately. + +**Prerequisites** + +* [Docker](https://docs.docker.com/) +* [g-node/gin-web](https://hub.docker.com/r/gnode/gin-web) node running in a docker container. If you don't have this then folow steps until [#install-gin](#install-gin) + +It's advisable to use all the **gin micro-services** inside docker containers and further connect all containers to interact with each other for increased fault tolerance. + +
+ +**Steps** + + +**[1]** In order to allow all gin containers to interact with each other, create a **docker network** and connect all containers to this network. + +`docker network create ` + +**[2]** If you already have a **gin** service container running, then attach it to the new network. + +`docker network connect ` + + +**Or** if you do not have a **gin** service container running already, then start a new one with the following command. +To keep things easier, we'll attach a static IP `172.19.0.2` to this container so that we don't have to inspect the docker network for changes in IP later. + +`docker run --name=gin --net --ip 172.19.0.2 -p 10022:22 -p 3000:3000 -v /var/gogs:/data gnode/gin-web:rebased` + +If you already had a gin container running, then run the following command to check and copy IP address of **gin** service container. + +`docker network inspect ` + +Copy the IP of your GIN container from `containers` section. + +**[3]** Set-up GIN Service. + +If you have already set-up a gin container previuosly, then create a custom configuration file ([check GOGS docs for custom config](https://gogs.io/docs/features/custom_template)) with the following details. + + +``` +DOMAIN = 172.19.0.2 +HTTP_PORT = 3000 +ROOT_URL = http://172.19.0.2:3000/ +``` + +Keep everything else the same. + +If you have set-up the gin service for the first time through this tutorial, then access `172.19.0.2:3000` in your browser. You will be treated with the configuration page. + +Mention the same config details as [above](#config) and leave everything else the same. + +Save the details. Register for a new account and login. + +**[4]** Create your first repository on the gin service. Clone the repository anywhere on your machine. + +**[5]** Create **drone** CI/CD service container. + +``` +docker run --volume=/var/run/docker.sock:/var/run/docker.sock --volume=/gin-proc/cache:/cache --env=DRONE_GIT_ALWAYS_AUTH=false --env=DRONE_GOGS_SERVER=http://172.19.0.2:3000 --env=DRONE_RUNNER_CAPACITY=2 --env=DRONE_RUNNER_NETWORKS=gin --env=DRONE_SERVER_HOST=172.19.0.3 --env=DRONE_SERVER_PROTO=http --env=DRONE_TLS_AUTOCERT=false --publish=80:80 --publish=443:443 --publish=2224:22 --restart=always --detach=true --net gin --name=drone drone/drone:latest + ``` + +Access drone at `172.19.0.3:80` in your browser. + +**[6]** Create **gin-proc** service. + +``` +docker run --net gin --name=gin-proc gnode/proc: +``` + +**Additional Environment Variables** + +`DEBUG=True` - Serves log messages from the lowest DEBUG level i.e. DEBUG, INFO, WARNING, ERROR, EXCEPTION. + +`DEBUG=False` or simply, not assigning the var - Servers log messages from and above INFO level i.e. INFO, WARNING, ERROR, EXCEPTION. + +`LOG_DIR=/path/to/your/log/dir` - Specific location to store your logs in. Otherwise, the logs are simply printed on your console in real time. diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 00000000..5751afe0 --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,55 @@ + +## Operations + + +#### What happens just after you Log-In + +To read API operations after `login`, please visit: `:8000/docs/api` + +
+ + +#### What happens after you run the workflow from front-end + +Your input data from the front-end is sent to the REST API running on port `8000` on the same address. And the API takes over the job from there. + +Read API Documentation for more info at: `:8000/docs/api` + +
+ + +### Inside the Pipeline + +The Drone config `gin-proc` service writes for you essentially does the following inside Drone containers (/runners) in chronological order: + +**+** Run a check whether you have run any previous builds on the repository in question, and if yes - then restore the repo from cache in order to speed up the clone process afterwards. + +**+** Starts the SSH agent. + +**+** Creates an SSH direcotory and writes your `priv-key` from Drone secrets of that repository to that directory, for Drone's use to clone your repository from GIN afterwards. + +**+** Disables `Strict Host Checking` for all authorized keys. + +**+** Adds your newly written `priv-key` to the SSH agent. + +**+** Runs a keyscan on your GIN container and writes the returned SSH fingerprint to `authorized_keys` + +**+** Clones your repository from GIN. + +**+** Installs Python dependencies if a `requirements.txt` file exists in your repo. + +**+** Annex your files if you had mentioned any files to be annexed - after initialising an annex repo in the working directory. + +**+** Attaches the commands or runs the Snakefile depending on your workflow chosen. + +**+** Add the backpush files, if you added any, and push them back to your repository in a separate `gin-proc` branch after completion of build job. + +**+** Pushes the produced output files back finally. + +**+** Rebuilds the latest cache of the repo after final operations. + +**+** Integrates the required volumes := `/cache` and `/repo` + +**+** Tells Drone to only run the build job in the event if you **pushed** to the **master** branch of your repository. Not in any otherwise case. + + **+** Attaches Slack or Email notification plugin if you had selected it from the front-end. \ No newline at end of file diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 00000000..8a7f4797 --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,18 @@ + +## Usage + +Go to your `gin-proc` address on port 3000 to access the service UI. Only use your GIN credentials to login. Users are not required to signup separately for `gin-proc`. + + +### Types of Workflows + +**Snakemake** - If you choose this workflow, you can enter the location of your Snakefile in your repo. If nothing is mentioned in location on during submission, then default location will be assumed to be the root of the repository. + +**Custom** - Users can enter the exact commands they want to run inside containers post cloning of the repository. Commands will be executed in the same exact order they are added to the workflow in. + + +### User's files + +**Annex Files** - Users can mention whether any files have to be especially `git-annex`ed before executing the workflow. + +**BackPush Files** - Users can mention the output files produced post execution of the workflow to be pushed back to user's GIN repository. Files will be pushed to a separate `gin-proc` branch in the repository. diff --git a/front-end/nuxt.config.js b/front-end/nuxt.config.js index e5b4e1ec..415b8a56 100644 --- a/front-end/nuxt.config.js +++ b/front-end/nuxt.config.js @@ -46,7 +46,7 @@ export default { ** See https://axios.nuxtjs.org/options */ axios: { - baseURL: 'http://localhost:8000' + baseURL: 'http://localhost:8000/auth' }, auth: { strategies: { diff --git a/front-end/pages/index.vue b/front-end/pages/index.vue index 394c4475..1c7ac28c 100644 --- a/front-end/pages/index.vue +++ b/front-end/pages/index.vue @@ -160,7 +160,7 @@