Skip to content

Commit

Permalink
docs: update latest docs with install / prod details
Browse files Browse the repository at this point in the history
  • Loading branch information
@spwoodcock
spwoodcock committed Nov 9, 2023
1 parent 2a4fdb6 commit ceb02a6
Show file tree
Hide file tree
Showing 3 changed files with 150 additions and 129 deletions.
263 changes: 136 additions & 127 deletions INSTALL.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,41 +5,39 @@
Use the provided bash script:

```bash
curl -fsSL https://get.fmtm.dev | bash
# Download the script
curl -L https://get.fmtm.dev -o install.sh
# OR (alternative URL)
curl -L https://fmtm.hotosm.org/install.sh

# OR

curl -fsSL https://fmtm.hotosm.org/install.sh | bash
# Run the script
bash install.sh

# Then follow the prompts
```

> Note: it is best to run this script as a user other than root.
> However, if you run as root, a user svcfmtm will be created for you.
## Manual Way

> NOTE: This is an installation guide to quickly get the fmtm app up and running.
> For a detailed guide on how to install the fmtm app using different
> methods and contributing, checkout the
> [dev docs](https://hotosm.github.io/fmtm/dev/Setup/)
If more details are required, check out the
[dev docs](https://hotosm.github.io/fmtm/dev/Setup/)

# Table of Contents
### Table of Contents

1. [Software Requirements](#software-requirements)

2. [Setting up the Backend](#setting-up-the-backend)
2. [Setting up FMTM](#setting-up-fmtm)

- [Fork and Clone the FMTM repository](#fork-and-clone-the-fmtm-repository)
- [Development: Setup Your Local Environment](#development-setup-your-local-environment)
- [Development: Setup Your Local Environment](#setup-your-local-environment)
- [Start the API with Docker](#start-the-api-with-docker)
- [Setup ODK Central User](#setup-odk-central-user)
- [Import Test Data](#import-test-data)
- [Check Authentication](#check-authentication)

3. [Setting up the Frontend](#setting-up-the-frontend)
- [Setup ODK Central User (Optional)](#setup-odk-central-user-optional)
- [Import Test Data (Optional)](#import-test-data-optional)
- [Check Authentication (Optional)](#check-authentication-optional)

- [Fork and Clone the FMTM repository](#fork-and-clone-the-fmtm-repository-1)
- [Start the Frontends with Docker](#start-the-frontends-with-docker)

# 1. Software Requirements <a name="software-requirements"></a>
## Software Requirements

Before you can install and use this application,
you will need to have the following software
Expand All @@ -48,181 +46,192 @@ installed and configured on your system:
- [Git](https://git-scm.com/)
- [Docker](https://docs.docker.com/)

To install Git, please follow the instructions on the official Git website:

https://git-scm.com/downloads

To install Docker, please follow the instructions on the official Docker website:
To install Git, please follow the instructions on the
[official Git website](https://git-scm.com/downloads)

https://docs.docker.com/engine/install/
To install Docker, please follow the instructions on the
[official Docker website](https://docs.docker.com/engine/install/)

# 2. Setting up the Backend <a name="setting-up-the-backend"></a>
## Setting up FMTM

## Fork and Clone the FMTM repository
### Fork and Clone the FMTM repository

### 1. Fork the repository
#### 1. Fork the repository

Forking creates a copy of the repository in your own GitHub account.
Go to the [Field Mapping Tasking Manager repository](https://github.com/hotosm/fmtm) and click the "Fork" button in the top right corner of the page.
Go to the [Field Mapping Tasking Manager repository](https://github.com/hotosm/fmtm)
and click the "Fork" button in the top right corner of the page.

### 2. Clone the forked repository
#### 2. Clone the forked repository

Clone the forked repository to your local machine using the following command:

`git clone https://github.com/<your-username>/fmtm.git`
```bash
git clone https://github.com/<your-username>/fmtm.git

# If you wish to deploy for production, change to the main branch
git checkout main
```

Make sure to replace `<your-username>` with your GitHub username.

## Development: Setup Your Local Environment
### Setup Your Local Environment

These steps are essential to run and test your code!

### 1. Setup OSM OAUTH 2.0
#### 1. Setup OSM OAUTH 2.0

The FMTM uses OAUTH2 with OSM to authenticate users. To properly configure your FMTM project, you will need to create keys for OSM.
The FMTM uses OAUTH2 with OSM to authenticate users.

1. [Login to OSM](https://www.openstreetmap.org/login) (_If you do not have an account yet, click the signup button at the top navigation bar to create one_). Click the drop down arrow on the extreme right of the navigation bar and select My Settings.
To properly configure your FMTM project, you will need to create keys for OSM.

2. Register your FMTM instance to OAuth 2 applications. Put your login redirect url as `http://127.0.0.1:7051/osmauth/`, For Production replace the URL as production API Url
1. [Login to OSM](https://www.openstreetmap.org/login)
(_If you do not have an account yet, click the signup
button at the top navigation bar to create one_).
Click the drop down arrow on the top right of the navigation bar
and select My Settings.

> Note: `127.0.0.1` is required instead of `localhost` due to OSM restrictions.
2. Register your FMTM instance to OAuth 2 applications.
Put your login redirect url as `http://127.0.0.1:7051/osmauth/` if running locally,
or for production replace with https://{YOUR_DOMAIN}/osmauth/

<img width="716" alt="image" src="https://user-images.githubusercontent.com/36752999/216319298-1444a62f-ba6b-4439-bb4f-2075fdf03291.png">
> Note: `127.0.0.1` is required for debugging instead of `localhost`
> due to OSM restrictions.
3. Right now _read user preferences permission_ is enough later on fmtm may need permission to modify the map option which should be updated on OSM_SCOPE variable on .env , Keep read_prefs for now.
![image](https://user-images.githubusercontent.com/36752999/216319298-1444a62f-ba6b-4439-bb4f-2075fdf03291.png)

4. Now Copy your Client ID and Client Secret. Put them in the `OSM_CLIENT_ID` and `OSM_CLIENT_SECRET` of your `.env` file
3. Only the _read user preferences permission_ is required as of now.

### 2. Create an `.env` File
4. Now save your Client ID and Client Secret for the next step.

Environmental variables are used throughout this project. To get started, create `.env` file in the top level dir, a sample is located at `.env.example`
#### 2. Create an `.env` File

cp .env.example .env
Environmental variables are used throughout this project.
To get started, create `.env` file in the top level dir,
a sample is located at `.env.example`.

Your env should look like this:
This can be created interactively by running:

```dotenv
### ODK Central ###
ODK_CENTRAL_URL=https://proxy
ODK_CENTRAL_USER=`<any_valid_email_address>`
ODK_CENTRAL_PASSWD=`<password_of_central_user>`
```bash
bash scripts/gen-env.sh
```

### FMTM ###
# DEBUG=True
# LOG_LEVEL=DEBUG
FMTM_DOMAIN=fmtm.localhost
### Start the API with Docker

### OSM ###
OSM_CLIENT_ID=`<OSM_CLIENT_ID_FROM_ABOVE>`
OSM_CLIENT_SECRET=`<OSM_CLIENT_SECRET_FROM_ABOVE>`
OSM_URL=https://www.openstreetmap.org
OSM_SCOPE=read_prefs
OSM_LOGIN_REDIRECT_URI=http://127.0.0.1:7051/osmauth/
OSM_SECRET_KEY=<random_key_for_development>
This is the easiest way to get started with FMTM.

### S3 File Storage ###
S3_ENDPOINT="http://s3:9000"
S3_ACCESS_KEY=`<a_long_access_key>`
S3_SECRET_KEY=`<a_long_secret_key>`
Docker runs each service inside **containers**, fully isolated from your
host operating system.

### Database (optional) ###
CENTRAL_DB_HOST=central-db
CENTRAL_DB_USER=odk
CENTRAL_DB_PASSWORD=odk
CENTRAL_DB_NAME=odk
##### Prerequisite

FMTM_DB_HOST=fmtm-db
FMTM_DB_USER=fmtm
FMTM_DB_PASSWORD=fmtm
FMTM_DB_NAME=fmtm
```
You will need to [Install Docker](https://docs.docker.com/engine/install/)
and ensure that it is running on your local machine.

## Start the API with Docker
Then from the command line, navigate to the top level directory of the FMTM project.

The easiest way to get up and running is by using the FMTM Docker deployment. Docker creates a virtual environment, isolated from your computer's environment, installs all necessary dependencies, and creates a container for the database, the api, and the frontend. These containers talk to each other via the URLs defined in the docker-compose file and your env file.
##### Select the install type

### Starting the Containers
Determine the what type of FMTM install you would like:

1. You will need to [Install Docker](https://docs.docker.com/engine/install/) and ensure that it is running on your local machine.
2. From the command line, navigate to the top level directory of the FMTM project.
3. From the command line run: `docker compose pull`.
This will pull the latest container builds from **main** branch.
4. Once everything is pulled, from the command line run: `docker compose up -d api`
5. If everything goes well you should now be able to **navigate to the project in your browser:** `http://api.fmtm.localhost:7050/docs`
```text
main - the latest production
staging - the latest staging
development - the latest development (warning: may be unstable)
local test - used during development, or to start a test version
```

> Note: If those link doesn't work, check the logs with `docker logs fmtm_api`.
The corresponding docker-compose files are:

## Setup ODK Central User
```text
main - docker-compose.main.yml
staging - docker-compose.staging.yml
development - docker-compose.development.yml
local test - docker-compose.yml
```

The FMTM uses ODK Central to store ODK data.
Set your selection to a terminal variable to make the next step easier:

- By default, the docker setup includes a Central server.
- The credentials should have been provided in your `.env` file to automatically create a user.
- To create a user manually:
```bash
export GIT_BRANCH={your_selection}

# E.g.
export GIT_BRANCH=development
```

##### Pull the Images

```bash
docker compose exec central odk-cmd --email [email protected] user-create
docker-compose exec central odk-cmd --email [email protected] user-promote
docker compose -f "docker-compose.${GIT_BRANCH}.yml" pull
```

> Note: Alternatively, you may use an external Central server and user.
> This will pull the latest containers for the branch you selected.
## Import Test Data
##### Build the Frontend

Some test data is available to get started quickly.
Before we can run, you need to build your version of the frontend.

- Navigate to the `import-test-data` endpoint in the API docs page:
<http://api.fmtm.localhost:7050/docs#/debug/import_test_data_debug_import_test_data_get>
- Click `Try it out`, then `execute`.
This is because the frontend contains variable specific to your deployment.

## Check Authentication
```bash
docker compose -f "docker-compose.${GIT_BRANCH}.yml" build ui
```

Once you have deployed, you will need to check that you can properly authenticate.
##### Start the Containers

1. Navigate to `http://api.fmtm.localhost/docs`
```bash
docker compose -f "docker-compose.${GIT_BRANCH}.yml" up -d
```

Three endpoints are responsible for oauth
<img width="698" alt="image" src="./images/endpoints_responsible_for_auth_screenshot-2023-03-26-092756.png">
You should see the containers start up in order.

2. Hit `/auth/osm_login/` : This will give you the Login URL where you can supply your osm username/password
Once complete, you should now be able to **navigate to the project in your browser:**

Response should be like this :
```text
https://{YOUR_DOMAIN}
{"login_url": "https://www.openstreetmap.org/oauth2/authorize/?response_type=code&client_id=xxxx"}
# For the local test setup, this will be
http://fmtm.localhost:7050
```

Now Copy your login_url and hit it in new tab, and you will be redirected to OSM for your LOGIN. Give FMTM the necessary permission
> Note: If those link doesn't work, check the logs with `docker logs fmtm-api`.
> Note: Use `docker ps` to view all container names.
After successful login, you will get your `access_token` for FMTM Copy it and now you can use it for rest of the endpoints that need authorizations
### Setup ODK Central User (Optional)

3. Check your access token: Hit `/auth/me/` and pass your `access_token` You should get your osm id, username and profile picture id
The FMTM uses ODK Central to store ODK data.

That's it, you have successfully set up the backend!!
- By default, the docker setup includes a Central server.
- The credentials should have been provided in your `.env`
file to automatically create a user.
- To create a user manually:

# 3. Setting up the Frontend <a name="setting-up-the-frontend"></a>
```bash
docker compose exec central odk-cmd --email [email protected] user-create
docker-compose exec central odk-cmd --email [email protected] user-promote
```

## Fork and Clone the FMTM repository
> Note: Alternatively, you may use an external Central server and user.
### 1. Fork the repository
### Import Test Data (Optional)

Forking creates a copy of the repository in your own GitHub account.
Go to the [Field Mapping Tasking Manager repository](https://github.com/hotosm/fmtm) and click the "Fork" button in the top right corner of the page.
If running a local test version, test data is available to get started quickly.

### 2. Clone the forked repository
- Navigate to the `import-test-data` endpoint in the API docs page:
<http://api.fmtm.localhost:7050/docs#/debug/import_test_data_debug_import_test_data_get>
- Click `Try it out`, then `execute`.

Clone the forked repository to your local machine using the following command:
### Check Authentication (Optional)

`git clone https://github.com/<your-username>/fmtm.git`
Once you have deployed, you will need to check that you can properly authenticate.

Make sure to replace `<your-username>` with your GitHub username.
1. Navigate to your frontend (e.g. `http://fmtm.localhost:7050`)

## Start the Frontend with Docker
2. Click the 'Sign In' button and follow the popup prompts.

1. You will need to [Install Docker](https://docs.docker.com/engine/install/) and ensure that it is running on your local machine.
2. From the command line: navigate to the top level directory of the FMTM project.
3. From the command line run: `docker compose build ui`
This is essential, as the development container for the frontend is different to production.
4. Once everything is built, from the command line run: `docker compose up -d ui`
3. If successful, you should see your username in the header.

5. If everything goes well you should now be able to **navigate to the project in your browser:** <http://fmtm.localhost:7051>
4. If you see an error instead, double check your credentials and
redirect URL in the openstreetmap.org settings.

That's it, you have successfully set up the frontend!!
That's it, you have successfully set up FMTM!!
2 changes: 1 addition & 1 deletion docs/dev/Backend.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The easiest way to get up and running is by using the FMTM Docker deployment. Do
5. Once everything is pulled, from the command line run: `docker compose up -d api`
6. If everything goes well you should now be able to **navigate to the project in your browser:** `http://api.fmtm.localhost:7050/docs`

> Note: If that link doesn't work, check the logs with `docker log fmtm_api`.
> Note: If that link doesn't work, check the logs with `docker log fmtm-api`.
> Note: the database host `fmtm-db` is automatically resolved by docker compose to the database container IP.
Expand Down
14 changes: 13 additions & 1 deletion docs/dev/Production.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,19 @@ runs, and you can access the working website from the domain name!
- To access the database via GUI tool such as PGAdmin, it is possible using port tunneling.

```bash
ssh [email protected] -N -f -L 5430:localhost:5432
ssh [email protected] -N -f -L {local_port}:localhost:{remote_port}

# Example
ssh [email protected] -N -f -L 5430:localhost:5433
```

This will map port 5432 on the remote machine to port 5430 on your local machine.

## Manual Database Backups

```bash
backup_filename="fmtm-db-backup-$(date +'%Y-%m-%d').sql.gz"
echo $backup_filename

docker exec -i -e PGPASSWORD=PASSWORD_HERE fmtm_db pg_dump --verbose --format c -U fmtm fmtm | gzip -9 > "$backup_filename"
```

0 comments on commit ceb02a6

Please sign in to comment.