Datasets

General Information

The datasets listed below are all preloaded into the HackReduce hadoop clusters and ready for immediate use at the event. The [datasets/*] notice next to each title indicates the path where its located depending on where you want to access it:

• Hadoop HDFS: Can be found at /datasets/*
• Namenode local filesystem: Can be found at /mnt/datasets/*
• HackReduce Github project: Samples found in the datasets/* folder of the project

There's also the possibility of loading new data at the event, but this process could take a few hours. Please see one of the Hopper event organizers (probably Greg) about loading new data into your clusters.

Million Song Dataset [datasets/msd]

• http://labrosa.ee.columbia.edu/millionsong/pages/getting-dataset

Special thanks to Echo Nest for converting the whole 200+ GB HDF5 format of the dataset to TSV for us

NASDAQ daily prices and dividends [datasets/nasdaq]

• http://www.infochimps.com/datasets/daily-1970-2010-open-close-hi-low-and-volume-nasdaq-exchange

NYSE daily prices and dividends [datasets/nyse]

• http://www.infochimps.com/datasets/daily-1970-2010-open-close-hi-low-and-volume-nyse-exchange

Wikipedia XML dump [datasets/wikipedia]

• http://en.wikipedia.org/wiki/Wikipedia:Database_download#English-language_Wikipedia

Google Ngram [datasets/ngrams]

• Only the 1-gram and 2-gram datasets are available
• http://ngrams.googlelabs.com/datasets

Geonames [datasets/geonames]

• http://download.geonames.org/export/dump/readme.txt

Reddit voting data [datasets/reddit]

• http://www.reddit.com/r/redditdev/comments/dtg4j/want_to_help_reddit_build_a_recommender_a_public/

Bixi Montreal [datasets/bixi]

• XML dump of all the bike station information queried every minute over a couple of months.
• Provided by Fabrice

DNS dataset [datasets/dns]

• Contains the root file with all the domain names and their associated nameservers for the "com" TLD.

LDEO Surface Ocean CO2 Climatology data [datasets/ldeo]

• http://www.pmel.noaa.gov/co2/story/LDEO+Surface+Ocean+CO2+Climatology

Twitter dataset [datasets/twitter]

• Data of the social graph, user id to names, and selected celebrity profiles. This does not contain actual tweets because of Twitter policies.
• http://an.kaist.ac.kr/traces/WWW2010.html

Flight dataset [datasets/flights]

• Limited set of flight data containing origin, destination, departure time, return time, price and date. Only has flights originated from SEA.
• Provided by Hopper

Amazon dataset [datasets/amazon]

• Description of data formats: http://131.193.40.52/data/README.txt
• Data listing: http://131.193.40.52/data/

IMDB dataset [datasets/imdb]

• ftp://ftp.fu-berlin.de/pub/misc/movies/database/

Taylor Tweets [datasets/taylor_tweets]

• Taken around of the time of Elizabeth Taylor's death in late March 2011, this dataset consists was a search of all tweets containing the word "taylor" in them.
• JSON format