Re:Archiving web sites
by megous on Friday November 23, 2018 @07:57AM
(#18516110)
Communication is dependent on JS events, which are dependent on user's action. There's also localStorage and other such things. Your method might work for some simple JS based websites, but it's no silver bullet.
[ Parent
| on HN ]
While what you say is true, the above method is the only method to archive arbitrary web pages. Yes it depends on user interactions to some extent, but it's possible to reasonably let a page load until fetches stop, and consider it rendered. Generally speaking, you can only archive some preset interactions with a modern web page. You can't hope to capture it all.
There are tools like WebRecorder[0] that do this to some extent by recording and replaying all requests. It's certainly a step in the right direction and demonstrates that the approach is viable. This was the only approach I tried that worked for archiving stuff like three.js demos. Worth mentioning there's also an Awesome list[1] that covers various web archival technologies.
Re:Archiving web sites
by nikisweeting on Friday November 23, 2018 @08:20PM
(#18519879)
What I'm trying to do with Bookmark Archiver is record user activity to create a puppeteer script while the user is visiting the page, then replay that on a headless browser later and record the session to a WARC. That should cover both dynamically requested and interactive stuff that is otherwise being missed by current archiving methods.
I also plan on saving an x86 VM image of the browser used to make the archive every couple months so that sites can be replayed many decades into the future.