River Utilities

Jump to bottom Edit New page

Matthew Taylor edited this page Sep 11, 2015 · 1 revision

Twitter Scraper

If your data source is a Twitter account page, there is already a utility to scrape the HTML and return tweet objects.

var twitterScraper = require('../lib/twitter-scraper');

module.exports = function(body, options, temporalDataCallback, metaDataCallback) {
    twitterScraper(body, function(err, tweets) {
        
    });
};

Zipped Payloads

If your data URLs lead to a zip or gz file, there are utilities for your to unzip them.

var riverUtils = require('../../lib/river-utilities');

// For zip files:
riverUtils.zippedPathToString(downloadUrl, function(err, contents) {

});

// For gzip files:
riverUtils.gZippedPathToString(stationsUrl, function(error, resp, body) {

});