-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
update readme, alembic revision script
- Loading branch information
Showing
13 changed files
with
104 additions
and
2,204 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,90 +1,107 @@ | ||
| # Omotes REST service | ||
|
|
||
| This webservice allows the MapEditor to call the Omotes SDK to submit new workflow jobs, see their status and retrieve | ||
| This webservice allows the MapEditor to call the Omotes SDK to submit new workflow jobs, see their | ||
| status and retrieve | ||
| the results. | ||
|
|
||
| The webservice is based on TNOs Flask REST API Template. | ||
|
|
||
| ## Usage | ||
|
|
||
| Copy `.env-template` to `.env` and fill with the appropriate values. | ||
| Copy `.env.template` to `.env` and fill with the appropriate values. | ||
| To set up the components (for windows run in `Git Bash`): | ||
| ``` | ||
|
|
||
| ```bash | ||
| ./scripts/setup.sh | ||
| ``` | ||
|
|
||
| To start the components: | ||
| ``` | ||
|
|
||
| ```bash | ||
| ./scripts/start.sh | ||
| ``` | ||
|
|
||
| To stop the components: | ||
| ``` | ||
|
|
||
| ```bash | ||
| ./scripts/stop.sh | ||
| ``` | ||
|
|
||
| # Flask REST API Template | ||
|
|
||
| This is a skeleton application for a REST API. It contains a modular setup that should prevent annoying circular imports | ||
| that are sometimes an issue when scaling up Flask applications. It also contains a | ||
| generic library of code developed in other projects (under tno/shared). | ||
|
|
||
| The key dependencies are: | ||
|
|
||
| - [Flask-smorest](https://flask-smorest.readthedocs.io): A REST API framework built on top | ||
| of [marshmallow](https://marshmallow.readthedocs.io/). | ||
|
|
||
| ## Using the application | ||
| ### Optional start with local code and exposed postgres (dev mode) | ||
|
|
||
| The application can run in Docker or locally. There is a docker-compose.dev.yml file which is meant for infrastructure | ||
| services, which are not part of your application. The purpose is to run that always, and then you can either use the | ||
| docker-compose.yml to start the API, or run it locally. | ||
| There is a dev mode for start which will use the local code for `omotes-rest`, `omotes-sdk-python` | ||
| and `omotes-sdk-protocol` (assuming the repos are all in the same folder as this `omotes-rest` | ||
| repo), instead of released docker images, and it will expose the postgres port: | ||
|
|
||
| The Makefile contains a number of common commands. To get started, you can run `make dev` and it should build the Docker | ||
| image and start the API in Gunicorn with the automatic reloader. If you wish to use the Flask local dev server, either | ||
| modify the docker-compose.yml file or start it outside of Docker, for example with `make dev-local`. | ||
|
|
||
| Before running the application locally though, you are advised to create a virtual environment. Install the dependencies | ||
| using `pip install -r requirements.txt`, or `make requirements`. Then, copy the .env-template file to .env. | ||
|
|
||
| Either way, the API should start on http://localhost:9200. Access the autogenerated docs | ||
| through http://localhost:9200/openapi or http://localhost:9200/redoc. | ||
|
|
||
| ## How to use this template | ||
|
|
||
| You would typically copy this template, and replace all instances of "flask_rest_api" with the name of your application. | ||
| Application specific code would then go under `tno/<your_application_name>/`. Adding code under `tno/shared` is of | ||
| course also possible, but keep in mind that the goal of that folder is to allow for sharing between multiple | ||
| repositories. A proper way to actually set that up still needs to be figured out though. | ||
|
|
||
| Your endpoints would go under `tno/<your_application_name/apis`, grouped by file. Don't forget to register your | ||
| blueprints in `tno/<your_application_name>/__init__.py`. | ||
|
|
||
| ## Notable features | ||
|
|
||
| There is a very permissive setup of CORS, so that an arbitrary frontend can perform requests to this REST PAI. | ||
|
|
||
| The application contains a setup of structlog, a structured logging framework. This makes it trivial to perform more | ||
| advanced logging, and will by default output JSON logs in production and colored tab-separated logs in development. | ||
|
|
||
| For dependency management we use pip-tools, which is a combination of pip-compile (which generates a requirements.txt | ||
| from a requirements.in file) and pip-sync (which synchronizes your virtualenv to the exact state as specified in the | ||
| requirements.txt). | ||
|
|
||
| There is a basic configuration of mypy (see mypy.ini) for static type checking. | ||
| ```bash | ||
| ./scripts/start-dev.sh | ||
| ``` | ||
|
|
||
| ## Design decisions | ||
| # Directory structure | ||
|
|
||
| The following directory structure is used: | ||
|
|
||
| - `ci/`: Contains all CI & other development scripts to help standardize the development workflow | ||
| for Linux. | ||
| - `config/`: Contains orchestrator workflow definitions configuration. The `workflow_config.json` | ||
| file will be overwritten by a volume mount when deploying via docker. | ||
| - `scripts/`: Setup, start en stop scripts. | ||
| - `src/`: Source code for omotes-rest. | ||
| - `unit_test/`: All unit tests for omotes-rest. | ||
| - `.dockerignore`: Contains all files and directories which should not be available while building | ||
| the docker image. | ||
| - `.env.template`: Template `.env` file to run the orchestrator locally outside of docker. | ||
| - `.gitignore`: Contains all files and directories which are not kept in Git source control. | ||
| - `dev-requirements.txt`: Pinned versions of all development and non-development dependencies. | ||
| - `Dockerfile`: The build instructions for building the docker image. | ||
| - `dev.Dockerfile`: Used when running or testing with local code from the `omotes-system` | ||
| repository. | ||
| - `pyproject.toml`: The Python project (meta) information. | ||
| - `requirements.txt`: Pinned versions of all dependencies needed to run the orchestrator. | ||
|
|
||
| # Development workflow | ||
|
|
||
| The scripts under `ci/` are used to standardize the development proces. | ||
|
|
||
| - `create_venv`: Creates a local virtual environment (`.venv/`) in which all dependencies may be | ||
| installed. | ||
| - `install_dependencies`: Installs all development and non-development dependencies in the local | ||
| virtual environment. | ||
| - `lint`: Run the `flake8` to check for linting issues. | ||
| - `test_unit`: Run all unit tests under `unit_test/` using `pytest`. | ||
| - `typecheck`: Run `mypy` to check the type annotations and look for typing issues. | ||
| - `update_dependencies`: Update `dev-requirements.txt` and `requirements.txt` based on the | ||
| dependencies specified in `pyproject.toml` | ||
|
|
||
| A typical development workflow would be: | ||
|
|
||
| 1. create and configure `.env` from `.env.template` | ||
| 2. run `create_venv` | ||
| 3. run `install_dependencies`. | ||
| 4. develop or update the codebase according to the requirements... | ||
| 5. run `lint`, `test_unit`, and `typecheck` to check for code quality issues. | ||
|
|
||
| All these scripts are expected to run from the root of the repository. | ||
|
|
||
| ## How to work with alembic to make database revisions | ||
|
|
||
| First set up the development environment with `create_venv` and `install_dependencies`. Then you | ||
| can make the necessary changes to `src/omotes-rest/db_models/`. Finally, a new SQL schema | ||
| revision may be generated using `alembic` by running: | ||
| ```bash | ||
| ./scripts/db_models_generate_new_revision.sh "revision message" | ||
| ``` | ||
|
|
||
| ### Flask-smorest | ||
| All database revisions will be automatically applied when omotes-rest is started. | ||
|
|
||
| Flask-smorest is but one option for REST APIs in Flask. It seems currently the most up to date option, and added bonuses | ||
| are that it is relatively lightweight, building on top of existing paradigms from Flask (such as Blueprints) and | ||
| building on top of marshmallow for schema validation. | ||
| ## Direct Alembic control | ||
|
|
||
| Flask-Restless just generates REST-like API's for your database models. This is not really REST and can be quite | ||
| fragile. | ||
| In case more control is necessary, you can run the necessary alembic commands directly after | ||
| activating the virtual environment. | ||
|
|
||
| Flask-RESTPlus is pretty nice and is very similar to Flask-smorest. It is heavier than Flask-smorest though, inventing | ||
| more of their own paradigms on top of Flask. It is hardly maintained though. | ||
| First, change directory: `cd src/` | ||
|
|
||
| There is also Flask-RESTX, which is a fork of Flask-RESTPlus so is very similar. We've used it together with | ||
| flask-accepts, marshmallow, and marshmallow-dataclass for pretty nice results. However, Flask-smorest does not need any | ||
| of that because it just directly builds on top of marshmallow, so the end result is cleaner. | ||
| - Make a revision: `alembic revision --autogenerate -m "<some message>"` | ||
| - Perform all revisions: `alembic upgrade head` | ||
| - Downgrade to a revision: `alembic downgrade <revision>` (revision 'base' to | ||
| undo everything.) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.