air

"air" is a project to visualize live air quality data provided by the Tokyo Metropolitan Government. The main components of the project are:

• a scraper to extract air data from www.kankyo.metro.tokyo.jp
• a postgres database to store the data
• an express.js server to serve this data and other static files to the client
• a client app that interpolates the data and renders an animated wind map

An instance of "air" is available at http://air.nullschool.net. It is currently hosted by Amazon Web Services and fronted by CloudFlare.

"air" is a personal project I've used to learn javascript, node.js, when.js, postgres, D3 and browser programming. Some of the design decisions were made simply to try something new (e.g., postgres). Undoubtedly, other decisions were made from a lack of experience. Feedback welcome!

building and launching

Clone the project and install libraries from npm:

npm install

NOTE: you will need libpq to build pg. The libpq library was installed automatically by postgres on Mac OS X but required separate installation on AWS.

Install postgres and create a database, something like:

CREATE DATABASE air
  WITH OWNER = postgres
       ENCODING = 'UTF8'
       TABLESPACE = pg_default
       LC_COLLATE = 'en_US.UTF-8'
       LC_CTYPE = 'en_US.UTF-8'
       CONNECTION LIMIT = -1;

Launch the server:

node server.js <port> <postgres-connection-string> <air-data-url>

Example:

node server.js 8080 postgres://postgres:12345@localhost:5432/air <air-data-url>

Finally, point the browser at the server:

http://localhost:8080

implementation notes

Building this project required solutions to some interesting problems. Here are a few:

• Live air data is available as Shift_JIS encoded HTML. Node.js does not natively support Shift_JIS, so the iconv library is used to perform the conversion to UTF-8.
• Geographic data of Tokyo was sourced directly from the Ministry of Land, Infrastructure, Transport and Tourism, as an 80MB XML file. This data was transformed to a 300KB TopoJSON file, small enough for browsers to download and render as SVG with D3.
• Roughly 50 sampling stations provide hourly wind and pollutant data. Inverse Distance Weighting interpolation is used to construct a wind vector field that covers all of Tokyo. IDW produces strange artifacts and is considered obsolete, but it is very simple and was easy to extend to perform vector interpolation.
• The browser interpolates each point (x, y) using the n-closest sampling stations. To determine the n-closest neighbors, the client constructs a k-d tree, which greatly improves the performance.
• Pollutant visualization overlays use Thin Plate Spline interpolation. TPS is definitely the wrong method to use for natural phenomenon such as air pollutants, but it produces a smoother surface than IDW.
• The SVG map of Tokyo is overlaid with an HTML5 Canvas, where the animation is drawn. The animation renderer needs to know where the borders of Tokyo are rendered by the SVG engine, but this pixel-for-pixel information is difficult to obtain directly from the SVG elements. To workaround this problem, Tokyo's polygons are re-rendered to a detached Canvas element, and the Canvas' pixels operate as a mask to distinguish points that lie inside the map to those outside. The field mask occupies the red color channel, and the display mask is encoded in the green color channel.
• I used when.js in the browser because it was a fun experiment.

inspiration

The awesome wind map at hint.fm provided the main inspiration for this project. And the very nice D3 tutorial Let's Make a Map showed how easy it was to get started.