Saving fetch responses initiated from JavaScript

[bryanph]

This is probably out of scope of this project currently, but I figured maybe there's a solution for this anyway ;)

Basically I'm trying to save some pages where a bunch of JS is loaded which in turn loads some JSON files from within the JavaScript. Obviously SingleFile can't really be aware of these fetches as it would have to parse the JavaScript and figure out where the fetches occur. But I was wondering if there is some way to save these kinds of pages regardless?

An example site where this occurs: https://ncase.me/trust/

One possible approach I can think of might be to "record" fetch requests/responses while refreshing the page and then monkey patch the fetch and XMLHttpRequest to return the recorded responses whenever a matching request is encountered.

Are there any other web archiving tools that attempt to solve this problem?

[gildas-lormeau]

Indeed, we reach the limits of SingleFile at this stage. Have you looked at what is being done around the WARC format, like https://github.com/webrecorder/pywb? There is also the Web Bundles format (see https://web.dev/web-bundles/) which is supposed to solve this kind of problems but the implementation is not finished yet AFAIK.

[bryanph]

Yeah I did look into WARC a little but the tooling around it seems pretty fragmented and a bit of a pain to set up. If creating a WARC file could be simply done using an extension it would probably have more adoption as a format. Then there's also the problem of actually playing these formats, it doesn't look like there's an easy way to run them in a browser directly. You need at least some kind of proxy server set up to make it work (like pywb), although of course that's a valid option for many use cases. The tooling around it seems to be mostly aimed at proper archiving by date as in what the internet archive does. Not so much for everyday daily use. An extension that allows you to directly record and play WARC files from your browser would really help make it a more valid approach I think. Although I don't know whether the playing of WARC files would be even possible without some sort of proxy server. Therefore I thought maybe creating a recording/playback feature using something like SingleFile as a base might make that process easier/possible. Who knows I might give it a shot at some point 😄

Web-bundles looks interesting as a potential future solution to this, will keep my eye on that one.