Creating a River

Step One: Find a public data source on the internet

Find a URL that contains useful or interesting data. It must be public (no authentication). This data should change over time. An HTTP request is sent to each URL, and the response body is passed to your parser.

Step Two: Name it

Create a directory in /rivers and name it something unique. This is where all the rivers go.

Step Three: Write a parser

Write a JavaScript function called parse.js that parses the response body and extracts a stream of data over time. See an example parser for NYC Traffic data. The function looks like this:

module.exports = function(config, body, url, fieldCallback, propertyCallback) {
    // 1. parse the body
    // 2. call the callbacks with data
};

Step Four: Write a config

Put it in config.yml like this. Each URL in sources is called at the interval specified and the response body text is sent to your parser.