This is the parent project for a suite of applications which take time series data from an instrument, store as tagged, searchable, indexed files, and display on a publicly accessible website.
Here is the original briefing note
Researchers commonly use instruments which save their data in a convenient data format. CSV (comma seperated values) are the de facto standard. The advent of remote file stores, like Microsoft OneDrive make it easy for researchers to share this data between research groups. That's the raw data - but how about if you want to discuss or compare different sets of data? That means importing particular data to a a spreadsheet, wrangling and formatting and then distributing the files. Often by email.
What we wanted to do with this project was to make it much, much easier for that data to be communicated.
Here's what you can do with this project:
- Import the raw data from a CSV file in OneDrive (other file store providers can be accomodated)
- Tag the data to make it easily discoverable
- View: Pan, Zoom the data
- Add annotations to particular points in the data
- Compare: add another data set, offset it to compare features
- Make your data sharable, by adding a 'public' tag.
- Share a particular view of the data, as a URL
This project is open source; we welcome use, feedback and pull requests. The project will also feature in MMU CMDT's Professional Development module as a Live Project: Student teams will be adding features to their own forks, and submitting pull requests.
Trello Team
Parent Trello Board
The MongoDB database will store the runs, authentication, algorithms palettes and tags. To host the database a combination of Heroku and mLab will be used.
To host the database first a Heroku account is needed, sign up to Heroku
From the Heroku dashboard create a new application, name the app, and choose the appropriate region.
To install mLab MongoDb navigate to mongolab addons page and install, ensuring the add-on is added to the newly created app
Within the Resources tab for the newely created application if you see the mLab MongoDB add-on mLab has been succesfully installed.
Within the Resources tab for the database application click the mLab MongoDB add-on to navigate to the mLab dashboard.
Under the users tab add a new database user, you will need the database-username and database-password to connect.
To create the mongoDB database the mongo shell will be used.
Before creating the MongoDB database, first MongoDB must be installed locally. Install MongoDB Community Edition for the appropriate OS.
To start the mongo shell change directory to the <mongodb installation dir>
cd <mongodb installation dir>
Type ./bin/mongo to start mongo. To ensure mongodb installation:
mongo --version
alternatively add <mongodb installation dir>/bin to the PATH environment variable.
Connect to the mongodb server, mLab dashboard provides instruction on how to.
mongo <mongo server URI> -u <dbuser> -p <dbpassword>
Verify the database, the output should be the <database name>
db
To create the starting database template:
db.createCollection("runsCollection",{capped:false}); db.createCollection("tagsCollection",{capped:false});db.createCollection("algorithmsCollection",{capped:false}); db.createCollection("authenticationCollection",{capped:false}); db.createCollection("palettesCollection",{capped:false}); db.algorithmsCollection.insert({"name" : "default","parameters" : [ "T(Copper)", "T(Cell1)", "DAC"],"algorithm" : "return((T1-T2)/((DAC*DAC)/22))"}); db.palettesCollection.insert({"name" : "default","description" : "default colour palette","palette" : [ "#B2003F", "#00B20F","#FF005A"]});
for more information of the starting database template see mongodb-database-template.md
To setup OneDrive application
A OneDrive Developer account is needed
- Navigate to Microsoft Graph : https://developer.microsoft.com/en-us/graph/
- Click the My Apps login, located on the top bar when not signed in.
- Microsoft will prompt for a login
- Once logged in, click the Add An App button
- Name the application and create
Within a specific application :
- Take note of the Application Id (also known as Client Id)
- Under Platforms click the Add Platform button
- Choose the Web Option
- Check the Allow Implicit Flow checkbox
- Under Redirect URLs enter the applications callback, example [application_domain]/callback.html
- Under Microsoft Graph Permissions , remove any default permissions
- Add the Sites.Read.All Application Permission (not Delegated Permissions)
The single page web application (SPWA) uses Auth0 for authentication.
To use authentication first a Auth0 account is needed, sign up to Auth0
The SPWA uses Auth0 SINGLE PAGE APPLICATION application type.
On the Auth0 dashboard under the Applications section create a new application, picking the Single Page Web Applications application type.
Under settings for the newly created application add the domain of the SPWA within the Allowed Callbacks URLs, Allowed Web Origins and Allowed Origins text areas.
If the domain of the application is not known the text areas can be filled later
Under Advanced Setting, under Endpoints take note of the OAuth Endpoints, these will be used for the Browse API
On the Auth0 dashboard under the API section create a new API.
Take note of the API Audience
When importing runs from the OneDrive into the database the runs must be in an exact format.
<date-time>/
-----------T-Data/
-----------------data.txt
-----------Remarks/
-------------------Remark01.txt
- run folder name must be the date and time delimited by a '-' format: yyyymmdd-hhmmss.
- Temperature_Log must be a csv file dimilited by a space, with a minimum of two columns. Line 1 must be column names, line 2 must be column units
- Remarks must be a txt file. Line 1 must be Remark added after < seconds > seconds where < seconds > is the remarks position. Line 2 must be the remark discription.
see example dataformat
The single page web application (SPWA) has two user types casual and admin both, below describes each use case.
The casual user is user who is not logged in thus admin controls are disabled.
- none
- public runs are only visible
- searching runs
- viewing & sharing runs
- exporting runs
To search for a run: on the home page of the SPWA the user can search by tags/time/date within the seach bar. When searching by tags a auto-complete will show.
Once a run has been searched for, a single run can be viewed simply by clicking on it, to view multiple runs use the checkboxes and blue View button.
One the view page the graph can be zoomed in and out using the mouse wheel and panned using click and drag. To offset the active drag click and drag while holding shift.
Underneath the graph is the Column-Tag panel, from here run tags can be viewed and columns visibility can be toggled. To switch the active graph simply click on a visible column.
To lock an axis, when panning or offsetting, the lock buttons located at the end of each axis can be toggled.
To view an annotation description click on an annotation label
Once a run has been searched for, runs can be zipped and saved to the users local machine by using the checkboxes to select multiple runs and clicking the green Export button.
The admin user is a user who is logged in thus admin controls are enabled. The admin users features and uses are a superset of the casual user features and uses.
- A OneDrive account is needed to import runs into the database, sign up to OneDrive
- Logged into the SPWA
- private runs are visible
- deleting runs
- modifying runs
- importing runs
Once a run has been searched for, runs can be deleted from the database (runs cannot be deleted from OneDrive) by using the checkboxes to select multiple runs and clicking the red Delete button.
On the view page, to add a new annotation click the '+' button.
To change the annotation position click the down-wards arrow button on the annotation panel. Using the black downward arrow click and hold to reposition. To confirm the new position click the black square.
To edit the annotation description click the pencil icon on the annotation panel.
One the Column Tag panel switch to view tags. Click the pencil icon to edit the tags.
To import runs from a OneDrive account, click the import button. If not already logged in click the login button to login into OneDrive. To preview a run click the graph preview button, to import runs select the checkboxes and click confirm.
This project was funded via the Marloes Peeters Research Group and mentored by DigitalLabs@MMU as a DigitalLabs Summer Project. It is the work of Yusof Bandar.
