Skip to content

Latest commit

 

History

History
36 lines (22 loc) · 1.19 KB

README.md

File metadata and controls

36 lines (22 loc) · 1.19 KB
Raw

gs-map-the-collections

GS@HackTheDeep: Map the Collections

Approach

Full Stack end-to-end user experience - from importing the file, cleaning and enriching the data then finally visualising it with full search capability

Tech Stack

  • Flask python framework
  • Worms API for taxonomy look up
  • Google Maps API for visualistion and geolocation look up
  • Python autocorrect library for spell checks
  • ElasticSearch to store the new transformed data
  • React.Js for a modular UI

Challenges

  • Taxonomy look up: special characters are currently ignored this causes more errors in the data than there actually could be
  • API calls needed to all be batched up
  • Run time for full transformation is currently quite lengthy
  • Learning ReactJS and other new libraries from scratch
  • Date format when purely text i.e. "Summer of 1992"

Future

  • create and extend the cleaning process to more columns
  • populate sparse location attributes by backtracking enriched geocoordinates
  • create a rest API to receive the "dirty" data file instead of manually uploading the file
  • add additional visualisation libraries on top of elasticSearch - now data stored in an easy query-able state