DBCA Turtle Data Viewer

Interactive turtle data preview for QA and analysis hosted at https://turtledata.dbca.wa.gov.au/ for DBCA staff.

Quickstart

• Open https://turtledata.dbca.wa.gov.au/, wait for data to load.
• Select an area of interest. You can bookmark your selection by appending ?site=Site Name, e.g. 
  • Delambre https://turtledata.dbca.wa.gov.au/?site=Delambre%20Island
  • Thevenard https://turtledata.dbca.wa.gov.au/?site=Thevenard%20Island
• Maps have a full-screen button and can toggle layers on and off. Individual markers are clustered for performance - click clusters or zoom in to expand them. Individual markers have popups with links to the original record on WAStD.
• (In development) A download button will generate a ZIP archive of all data currently shown. This replaces data dissemination via SharePoint. If you need fresh data, come and get it here.

Architecture

This section is for developers.

This Shiny app is based on bs4Dash and was initially created from a Golem template.

The main mechanism is to read snapshots of two fairly complex production databases and visualise the data through maps, figures, and summaries. The user can select areas of interest (mostly turtle monitoring program areas) and download the data for that area.

Data snapshots are locally saved as an .rds file for WAStD data, WAMTRAM data, and WAStD sites. The app accesses these snapshots through file paths relative to its working directory, e.g. here:here("inst/wastd_site.rds").

In development, the workdir is the top level folder of the app’s source code, and inst refers to that folder inside the source code.

In production, the WORKDIR is defined as /app in the Dockerfile, and data snapshots are written to /app/inst in the running container.

To persist the data snapshots, a volume should be mounted to the internal path /app/inst.

A second Docker image (dbca-wa/etlturtlenesting) runs the ETL jobs to periodically refresh the data snapshots from the production databases. The ETL process takes close to an hour, the saved snapshots read in seconds. The ETL includes import of all digitally captured data from ODK Central into WAStD prior to the data export from WAStD, so that all digitally captured data is visible in this dashboard within two hours of upload.

Development

• Write code, add tests where needed.
• Update dependencies from code: attachment::att_amend_desc() (careful: drops unused Imports).
• Update Dockerfile from dependencies: initially generated through golem::add_dockerfile(from="rocker/geospatial:4.1.2"). Note: rotate existing Dockerfile to Dockerfile_old to keep manual edits.
• Build Docker image for local testing (see below).
• Run e.g. through Portainer with
  • Name: e.g. “tv”
  • Image: advanced mode, ghcr.io/dbca-wa/turtleviewer2
  • Always pull image: off
  • Publish ports: on
  • Volumes: container path /app/inst, use named volume or bind mount to a local directory
  • env: paste environment variables from your .env (warning: sensitive credentials). A template is given at inst/.env.template.
• If the Docker image does not work, inspect the docker logs, or debug through the Docker shell. The console runs inside the Docker container, bypassing the Shiny app. There, you can run R and test whether all libraries (especially turtleviewer2) load successfully.
• If the app as well as the Docker image work, commit and push.

Run the app locally:

golem::detach_all_attached()
golem::document_and_reload()
run_app()

Build Docker image for local testing:

docker build -t ghcr.io/dbca-wa/turtleviewer2 .

Inspect the locally built image for debugging:

docker run -it ghcr.io/dbca-wa/turtleviewer2 /bin/bash -c "export TERM=xterm; exec bash"

Deployment

Once app and Docker image work, create a new version, tag, and push the tag.

styler::style_pkg()
spelling::spell_check_package()
spelling::update_wordlist()

# Code and docs tested, working, committed
usethis::use_version(which="patch")
usethis::use_version(which="minor")
usethis::use_version(which="major")
usethis::edit_file("NEWS.md")

# Document to load new package version. Git commit, tag, and push.
devtools::document()
v <- packageVersion("turtleviewer2")
system(glue::glue("git tag -a v{v} -m 'v{v}'"))
system(glue::glue("git push && git push --tags"))

Pushing a new tag will trigger a GitHub Action to build a new Docker image and publish it to the GitHub container registry.

In the Rancher console to the BCS Kubernetes cluster, edit the “turtleviewer2” workload and update the image to the latest tag. This action could be automated through the k3s kubectl shell.