Present + discuss at an IPFS community hangout?

[jbenet]

I'd be interested in discussing ipwb through a community hangout. and more generally "WARCs + IPFS"

cc @ikreymer of WebRecorder and @mekarpeles

[ibnesayeed]

Sure thing! We should be able to talk about it sometime next week. Currently we are wrapping our initial thoughts in a conference paper/poster and chasing the deadline.

[ikreymer]

Hi, sorry, I forgot to comment earlier as I've been quite busy. I think it'd be good for us to chat so that we're all on the same page.

We already know that we can store WARC data in ipfs, the question is how best to do it. I think that we should try to standardize on an approach for ingesting WARCs and exporting them back out out of IPFS. After our last chat, we were waiting on the IPLD implementation, but perhaps now is the time to revisit this.

For some background, there is this comment thread ipfs-inactive/archives#28 as the main thread for web archive related activities. There is also an implementation I created last year ikreymer/pywb-ipfs which provides a full browse, write WARC to IPFS, replay from IPFS solution. This was a demo with the full record being stored. This implementation provides a different way of storing works, separating the headers and the payload, which I think makes sense. I think the goal now is to iterate on this to make sure we know exactly how we want to proceed with storing WARCs.
One idea that was mentioned was even creating a standard ipfs warc add tool.

Once we have this figured out, I think we can then proceed to build more tools to use WARCs and IPFS. I already have many ideas, but am holding off until we have a standard approach. :)

[ikreymer]

I would suggest that those of us interested in this do a quick hangout/discussion before bringing it to a larger group? Let me know what days/times work for you.