Skip to content
/ server Public

Server software stack for NOAA's Integrated Ecosystem Assessment (IEA) program, containerized using Docker

License

MIT license
1 star 1 fork Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

noaa-iea/server

BranchesTags

Repository files navigation

instead on ecoquants.com

Add user

In RStudio:

# setup (once) staff to be shared by admin, and default permissions 775
sudo su
gpasswd -a admin staff
echo 'umask 002' >> /etc/profile

# override RStudio's default group read only with group read & write
echo 'Sys.umask('2')\n' >> /usr/local/lib/R/etc/Rprofile.site
# vs quick fix in Terminal of rstudio.marineenergy.app: sudo chmod -R g+w *

# Add shiny to staff so has permission to install libraries into `/usr/local/lib/R/site-library` and write files
usermod -aG staff shiny

# set primary group to staff
usermod -g staff shiny
#confirm primary group set to staff
id shiny
# uid=998(shiny) gid=50(staff) groups=50(staff)

user=kimberly.bastille
pass=ch@ngemE

# userdel $user; groupdel $user

# add user inside rstudio docker container from host
useradd -m -p $(openssl passwd -crypt $pass) $user
# echo usermod -p "$pass" $user
# usermod -p $(openssl passwd -crypt $pass) $user

# setup (every user) primary group to staff
usermod -aG staff $user
usermod -aG sudo $user
usermod -aG shiny $user
usermod -g staff $user
groups $user

# setup symbolic links in home dir
ln -s /share                /home/$user/share
ln -s /share/github         /home/$user/github
ln -s /srv/shinyapps        /home/$user/shiny-apps
ln -s /var/log/shiny-server /home/$user/shiny-logs

To change your password, in https://rstudio.ecoquants.com Terminal:

passwd

Add shiny app

Three easy steps:

# 1. git clone the repo
cd /share/github
git clone https://github.com/NOAA-EDAB/NEespShiny

# 2. serve shiny app
sudo ln -s /share/github/NEespShiny /srv/shiny-server/NEespShiny
  1. view it online: https://shiny.ecoquants.com/NEespShiny/

iea-server

Server software stack for MBON server serving dashboards Server Server

Contents:

Server software

Shell into server

  1. Connect to UCSB VPN via Secure Pulse
  2. SSH, eg for Ben: 
    sshpass -f ~/private/password_docker-iea-demo.us ssh [email protected]

Create Server on DigitalOcean

Created droplet at https://digitalocean.com with [email protected] (Google login):

  • Choose an image : Distributions : Marketplace :
    • Docker by DigitalOcean VERSION 18.06.1 OS Ubuntu 18.04
  • Choose a plan : Standard :
    • iea-demo.us:
      • $20 /mo $0.030 /hour
      • 4 GB / 2 CPUs
      • 80 GB SSD disk
      • 4 TB transfer
    • iea-demo.us:
      • $40 /mo $0.060 /hour
      • 8 GB / 4 CPUs
      • 160 GB SSD disk
      • 5 TB transfer
  • Choose a datacenter region :
    • San Francisco (New York currently experiencing issues)
  • Authentication :
    • One-time password Emails a one-time root password to you (less secure)
  • How many Droplets?
    • 1 Droplet
  • Choose a hostname :
    • iea-demo.us:
      • iea-demo.us

DigitalOcean - iea-ne.us project

Email recieved with IP and temporary password:

  • iea-demo.us:

    Your new Droplet is all set to go! You can access it using the following credentials:

    Droplet Name: docker-iea-demo.us IP Address: 157.245.189.38 Username: root Password: 513dbca94734429761db936640

Have to reset password upon first login.

Saved on my Mac to a local file:

ssh [email protected]
# enter password from above
# you will be asked to change it upon login
echo S3cr!tpw > ~/private/password_docker-iea-ne.us
cat ~/private/password_docker-iea-ne.us

Install Docker

Since we used an image with docker and docker-compose already installed, we can skip this step.

References:

sudo apt install apt-transport-https ca-certificates curl software-properties-common

# add the GPG key for the official Docker repository to your system
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# add the Docker repository to APT sources 
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"

# update the package database with the Docker packages from the newly added repo
sudo apt update

# install Docker
sudo apt install docker-ce

docker

# confirm architecture
uname -a
# Linux docker-iea-ne 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

# update packages
sudo apt update

# check that it’s running
sudo systemctl status docker

# add permissions to run docker for current user
sudo usermod -aG docker ${USER}

docker-compose

References:

# check for latest version at https://github.com/docker/compose/releases and update in url
sudo curl -L https://github.com/docker/compose/releases/download/1.25.5/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose

# set the permissions
sudo chmod +x /usr/local/bin/docker-compose

# verify that the installation was successful by checking the version:
docker-compose --version
# docker-compose version 1.25.4, build 8d51620a

Build containers

Test webserver

Reference:

docker run --name test-web -p 80:80 -d nginx

# confirm working
docker ps
curl http://localhost

returns:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Turn off:

docker stop test-web

Setup domain iea-ne.us

  • Bought domain iea-demo.us for $12/yr with account [email protected].

  • DNS matched to server IP 64.225.118.240 to domain iea-demo.us via Google Domains, plus the following subdomains added under Custom resource records with:

  • Type: A, Data:157.245.189.38 and Name:

    • @
    • wp
    • gs
    • rstudio
    • shiny
    • info
    • erddap
    • ckan

  • Name: www, Type: CNAME, Data:iea-ne.us

Run docker-compose

References:

First, you will create the environment .env file to specify password and host:

# get latest docker-compose files
git clone https://github.com/marinebon/iea-server.git
cd ~/iea-server

# set environment variables
echo "PASSWORD=S3cr!tpw" > .env
echo "HOST=mbon.marine.usf.edu" >> .env
cat .env

# launch
docker-compose up -d

# Creating network "iea-server_default" with the default driver
# Creating volume "iea-server_postgis-backups" with default driver
# Creating volume "iea-server_geoserver-data" with default driver
# Creating volume "iea-server_postgis-data" with default driver
# Creating volume "iea-server_mysql-data" with default driver
# Creating volume "iea-server_wordpress-html" with default driver
# Creating volume "iea-server_shiny-apps" with default driver
# Creating volume "iea-server_erddap-data" with default driver
# Creating volume "iea-server_erddap-config" with default driver
# Creating volume "iea-server_nginx-html" with default driver
# Pulling postgis (kartoza/postgis:11.0-2.5)...
# 11.0-2.5: Pulling from kartoza/postgis
# 68ced04f60ab: Pull complete
# ...

# OR update
git pull; docker-compose up -d

# OR build if Dockerfile updated in subfolder
git pull; docker-compose up --build -d

# git pull; docker-compose up -d --no-deps --build erddap

# OR reload
docker-compose restart

# OR stop
docker-compose stop

rstudio-shiny

Haven't figured out how to RUN these commands after user admin is created in rstudio-shiny container.

  1. Setup permissions and shortcuts for admin in rstudio.

    After logging into rstudio.iea-ne.us, to go to Terminal window and run:

    sudo su -
ln -s /srv/shiny-server /home/admin/shiny-apps
ln -s /var/log/shiny-server /home/admin/shiny-logs
mkdir /srv/github
ln -s /srv/github /home/admin/github

cd /srv/github
git clone https://github.com/marinebon/iea-ne_info.git
git clone https://github.com/marinebon/iea-uploader.git

chown -R admin /srv/shiny-server
chown -R admin /srv/github

ln -s /usr/share/nginx/html /home/admin/info-html
chown -R admin /home/admin/info-html

Docker maintenance

Push docker image

Since rstudio-shiny is a custom image bdbest/rstudio-shiny:s4w, I docker-compose push to bdbest/rstudio-shiny:s4w | Docker Hub.

# login to docker hub
docker login --username=bdbest

# push updated image
docker-compose push

Develop on local host

Note setting of HOST to local vs iea-ne.us:

# get latest docker-compose files
git clone https://github.com/marinebon/iea-server.git
cd ~/iea-server

# set environment variables
echo "PASSWORD=S3cr!tpw" > .env
echo "HOST=iea-ne.us" >> .env
cat .env

# launch
docker-compose up -d

# see all containers
docker ps -a

Then visit http://localhost or http://rstudio.localhost.

TODO: try migrating volumes in /var/lib/docker onto local machine.

Operate on all docker containers

# stop all running containers
docker stop $(docker ps -q)

# remove all containers
docker rm $(docker ps -aq)

# remove all image
docker rmi $(docker images -q)

# remove all volumes
docker volume rm $(docker volume ls -q)

# remove all stopped containers
docker container prune

Inspect docker logs

To tail the logs from the Docker containers in realtime, run:

docker-compose logs -f

docker inspect rstudio-shiny

shiny app shuffle

in rstudio.iea-demo.us terminal:

cd /srv
mkdir -p github/iea-ne_apps
cd /srv/shiny-server/
mv * ../github/iea-ne_apps/.
mv .git ../github/iea-ne_apps/.
mv .gitignore ../github/iea-ne_apps/.
ln -s /srv/github/iea-ne_apps/test /srv/shiny-server/test
cd ../github

ln -s /srv/github/iea-uploader /srv/shiny-server/uploader

erddap quick fix

git pull

nc_local=./erddap/data/iea-ne/ex-chl-ppd/M_201901-MODISA-NESGRID-CHLOR_A.nc
nc_docker=erddap:/erddapData/iea-ne/ex-chl-ppd/M_201901-MODISA-NESGRID-CHLOR_A.nc
docker exec erddap bash -c "mkdir -p /erddapData/iea-ne/ex-chl-ppd"
docker exec erddap bash -c "mkdir -p /usr/local/tomcat/conf/Catalina/localhost"
docker cp $nc_local $nc_docker

docker exec -it erddap bash -c "cd /usr/local/tomcat/webapps/erddap/WEB-INF && bash GenerateDatasetsXml.sh -verbose"

Doh! Still Bad Gateway at http://erddap.iea-demo.us/

Parameters for loading M_201901-MODISA-NESGRID-CHLOR_A.nc using:
GenerateDatasetsXml.sh -verbose

- Which EDDType: EDDGridFromNcFiles
- Parent directory: /erddapData/iea-ne/ex-chl-ppd
- File name regex: .*\.*nc
- Full file name of one file: /erddapData/iea-ne/ex-chl-ppd/M_201901-MODISA-NESGRID-CHLOR_A.nc
- ReloadEveryNMinutes: 10
- cacheFromUrl:
^D

TODO

Web content:

  • Rmd website served by nginx
  • infographics

Shiny apps:

  • data-uploader

Install:

About

Server software stack for NOAA's Integrated Ecosystem Assessment (IEA) program, containerized using Docker

Resources

Readme

License

MIT license
Activity
Custom properties

Stars

1 star

Watchers

7 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

No packages published

Languages