Overview

Repo for the True North data dashboard in partnership with Brabners.

Data pipelines

We use duckdb, python and R to pull data from EDD (which acts as a database) and other sources. This is the default and preferred option.

For static datasets, we download them manually into working.

We track data dependencies using dvc.

In pipelines we process the data sources to extract only the data we need for the site. These files are stored in data directories available to the website in src and checked in to GitHub. These data files drive the visualisations on the site.

Site structure

We use Lume to generate a static site and OI Lume charts for the visualisations.

Building the site

The site is built using the Javascript runtine deno, which will need to be installed. Installation instructions are available in the Deno Runtime getting started guide.

Once deno is installed, git clone the repo to the build server, and run

deno task build

This will create the static site assets in the _site directory. By default, this will be built with a server location as specified in the location option in _config.ts. This will affect the base path and any canonical URLs that are generated.

It is possible to override this when generating the site by setting the --location flag onthe build as follows:

deno task build --location https://www.brabners.com/true-north/data-explorer/

Bulding a release

The process for creating a release is to build the site, create a zip file and adding a new release to the GitHub repository.

This has been scripted as a Velociraptor action, which can be run from the command line as follows:

deno task vr make-release

This will mint a new release with the name set to a timestamp. WARNING There is no guarantee that the release differs from a prior release.

It is planned to place this in an action to automate the process.