Download whole site

[aolko]

Add a param to download whole site with assets, not just pages
(as of right now it only captures html page of the site)

[jsvine]

Thanks, @aolko! I worry slightly that a feature like this would lead to unintentionally massive downloads. On the other hand, I totally understand the impetus/desire to make archiving the assets easier. I'm going to give this a bit more thought, and then perhaps take a shot at prototyping it.

[MechMykl]

Wayback has copies of my old websites that I'd love to pull down in full. There are apps elsewhere to do this, so it makes sense to add the feature as an option for cases like these.

[jsvine]

Hi, @MechMykl. Makes sense. A couple of questions:

• Are you looking to, in one command, pull down every single version of your old website? Or are you looking to download specific snapshots? E.g., your entire website as it existed on Jan. 1, 2010?
• What are the other apps you mention? Would love to see how they handle this sort of thing.

[MechMykl]

I've used SiteSucker in the past to pull down copies of sites - http://www.macupdate.com/app/mac/11634/sitesucker

In one command, I'd love to pull down the first version of a website (complete with all HTML / images / etc.) and then only grab differences moving towards the most recent capture date. Ex:

My Website 2010 [HTML / Images / JS]
My Website 2011 [HTML]
My Website 2012 [Images]

This would help me better organize and review the years and years of backups Wayback has without having to navigate through broken records so often!

[Jolg42]

Indeed it would be great to download a whole website like some people did here https://raw.githubusercontent.com/typophile/typophile.github.io/master/wayback-typophile.pl to replicate a dead website.
In fact it doesn't look hard to do but I agree with you @jsvine it could have the same effect as a DDOS …

[jsvine]

Agreed; doesn't seem technologically very difficult. Because waybackpack downloads files sequentially (rather than in parallel), DDOS isn't so much a concern (unless I'm overlooking something) as massive bandwidth consumption. @wumpus: any thoughts on this?

[Jolg42]

@jsvine Sure sequential download is better for the servers 👍

By the way, here is a copy of Typophile's README & how they extract the uniques urls

Obtaining the HTML files

The Wayback Machine has a number of APIs, one of which is the CDX Server API.
This lists all URIs archived from a given site, and can be called on typophile like this:

curl 'http://web.archive.org/cdx/search/cdx?url=*.typophile.com&fl=urlkey,timestamp' > urls.txt ;

This will take a couple of minutes, and will return the complete list of all 471,100 URLs that it knows about.
Since there are 1,796 duplicates we can remove them:

awk '{print $1}' urls.txt | sort | uniq > urls-unique.txt ;

[wumpus]

We prefer one-at-a-time downloads. Even if you're downloading something big via our 60 gigabit link, it's reasonably fair to our other users thanks to the one-at-a-time-ness.

[MechMykl]

Saw that this got bumped to v0.3.4. Can't tell from the current readme, does Waybackpack pull down the site assets or just the linked HTML files? I'm excited to start grabbing my old sites but don't want to jump in yet if it's just the page code :)

[jsvine]

Thanks for checking in, @MechMykl. There's a nice pull request from @fgregg — #17 — that should provide a handy way to get all assets that match a wildcard search. One hitch: Unix filesystems are case-insensitive, while URL paths are not. (Try mkdir TMP; ls -la TMP; ls -la tmp;.) So the current approach — storing each asset at the path derived from its URL — breaks on wildcard searches that return URLs with differently-cased subpaths. (E.g., example.com/BIG_KAHUNA/photo.jpg and example.com/big_kahuna/photo.jpg). One possible solution would be to store assets in a SQLite database (or somesuch), but that'd take a little re-engineering.

[jwilk]

Unix filesystems are case-insensitive

Um, they are definitely case-sensitive:

$ touch foo
$ ls foo
foo
$ ls FOO
/bin/ls: cannot access 'FOO': No such file or directory

while URL paths are not

This is true, except for the percent-encoding triplets.
Source:
RFC 3986 §6.2.2.1
RFC 2616 §3.2.3

[jsvine]

Re. the case-sensitivity, I could have been clearer. Here's the issue:

• Let's say your website has pages at /mypath/index.html and /MyPath/index.html. These paths are case-sensitive, and point to different resources.

• Here's the problem you'd encounter on Unix:

$ mkdir mypath
$ touch mypath/index.html
$ mkdir MyPath
mkdir: MyPath: File exists

[n3storm]

@jsvine Are you using cygwin? because for sure Unix is not case-insensitive.

[jsvine]

Nope, I'm on OS X El Capitan. What machine are you on. And what happens when you run the commands in my previous comment?

[jwilk]

OK, so @jsvine's filesystem is most likely HFS+, which indeed is case-insensitive.
But this is not traditional Unix filesystem sematics.
Here, on Linux with ext4 fs, it is:

$ mkdir mypath
$ mkdir MyPath
$ ls
MyPath  mypath

[jsvine]

Many thanks, Jakub. That helps clarify the confusion on this thread; my apologies for my hand in it!

My gut feeling is that the best approach to waybackpacking a full site (and, perhaps, even for single pages) is to dump all resources into a database (SQLite by default, but anything that plays well with sqlalchemy). That'd make the filesystem a moot point, and also hypothetically enable some neat features — e.g., merging "packs" — and querying. What do you all think?

[Ninoninoninonino]

@jsvine Possibility for download of PDFs, .doc, .xls etc is really important for researchers. Maybe just ban photo and video, if at all?

[Ninoninoninonino]

@wumpus Could you please let us know more about Internet Archive policies on automated downloads?

[wumpus]

I no longer work at IA.

[JacobDB]

Any progress on this?

[jsvine]

Unfortunately, I haven't been able to carve out time for this yet. But it's certainly still on my todo list.

[EbuXa]

Hi jsvine,
when i want to download the spesific snapshot of whole pages of my site it only download the index.html file, the command is: waybackpack example.com -d downfolder/ --follow-redirects --from-date 201901

[lucky1804]

Any progress on this yet?

[lucky1804]

The website i want to download has images hosted on seperate CDN url, is it possible to get that too?

[jsvine]

Any progress on this yet?

Not currently.

The website i want to download has images hosted on seperate CDN url, is it possible to get that too?

Not currently.

[belisards]

Any progress on this? It would be great to follow the links recursively and download assets.

[jsvine]

Any progress on this?

Not currently, but you might have luck with the code in this PR: #17