News for hackers. Stuff that matters.
Re:Webrecorder: Capture interactive websites and replay them... by bsnnkv on Tuesday August 01, 2023 @12:53PM (#36958688)
It's ultimately a cat and mouse game with many websites actively trying to sabotage archival efforts.

Unless this is a core functionality in something you're working on, most people will be better off using the SavePageNow API from archive.org and integrating with that. This is what I ultimately ended up doing for one of my projects.[1]

[1]: https://lgug2z.com/articles/notado-07-2023-update/

Re:Webrecorder: Capture interactive websites and replay them... by ajvs on Tuesday August 01, 2023 @01:35PM (#36959398)
Website owners can request their site be blacklisted for archival, so this doesn't work for all websites.
Re:Webrecorder: Capture interactive websites and replay them... by linusg789 on Tuesday August 01, 2023 @01:48PM (#36959620)
Archive.today is another popular site that doesn't respond to blacklist requests.
Re:Webrecorder: Capture interactive websites and replay them... by bsnnkv on Tuesday August 01, 2023 @02:03PM (#36959860)
Yeah this is true, but it works for enough websites to be a meaningful option, and it is almost always going to work better than something you have home-rolled (unless your core product is a direct competitor or something, in which case all bets are off ;))

A good is example of this is Pinboard which claims to offer website archiving. A friend has over 100,000 links saved (with an archival account) there. When we spent a few minutes looking at the archive links for those items a few weeks ago, we couldn't find a single correct, working, accessible archive from the most recently archived links (listed as archived 5 weeks ago, so also not up to date).

Re:Webrecorder: Capture interactive websites and replay them... by e12e on Wednesday August 02, 2023 @04:56AM (#36968604)
I wonder if Firefox "reader mode as a utility" might be a viable alternative for Pinboard like "content oriented" archiving?

https://github.com/mozilla/readability