everarchive

A distributed, uncensorable web page archiver built on webtorrent, assetgraph & mongodb's gridfs

NOT FINISHED / USABLE YET

Current status/progress

Working (but probably needing cleanup)

• Crawling a page and saving assets to GridFS
• Creation of torrents containing all assets of archived page, via create-torrent
• Retrieval & serving of archived page from GridFS

In-progress

• Lookup of .torrent file for a given page via bitorrent-dht
• Seeding of created page torrents
  • This involves writing a new storage module for webtorrent, that does streamed I/O against Mongo's GridFS via gridfs-stream

To do

• Retrieval of page content from torrent
• Cleanup of crawler method
  • Will probably rework GridFS-related functionality into a writeAssetsToGridFs transform for assetgraph
• Shiny frontend (it's a literally just an unstyled text input & a submit button currently)
• Everything in TODO.md