Archiving web sites

Re:Archiving web sites by bicubic on Friday November 23, 2018 @08:15AM (#18516192)

While what you say is true, the above method is the only method to archive arbitrary web pages. Yes it depends on user interactions to some extent, but it's possible to reasonably let a page load until fetches stop, and consider it rendered. Generally speaking, you can only archive some preset interactions with a modern web page. You can't hope to capture it all.

There are tools like WebRecorder[0] that do this to some extent by recording and replaying all requests. It's certainly a step in the right direction and demonstrates that the approach is viable. This was the only approach I tried that worked for archiving stuff like three.js demos. Worth mentioning there's also an Awesome list[1] that covers various web archival technologies.

[0] https://github.com/webrecorder/webrecorder

[1] https://github.com/iipc/awesome-web-archiving

[ Parent | on HN ]

Re:Archiving web sites by nikisweeting on Friday November 23, 2018 @08:20PM (#18519879)

What I'm trying to do with Bookmark Archiver is record user activity to create a puppeteer script while the user is visiting the page, then replay that on a headless browser later and record the session to a WARC. That should cover both dynamically requested and interactive stuff that is otherwise being missed by current archiving methods.

I also plan on saving an x86 VM image of the browser used to make the archive every couple months so that sites can be replayed many decades into the future.

[ Parent | on HN ]