That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin and other needlessly flashy systems. I'm genuinely worried y'all are digging yourselves into a technical pit that you can't get out of and it will hurt or even kill your goals.
Cheers.
1) can consume a lot of storage really fast
2) makes IA a bot rather than a user service
3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.
I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserving the pages manually, and I don't imagine we'd even show up on ArchiveTeam's radar. But they're not large pages, and basically static. I don't think they'd be a huge burden to store and maintain. It seems like some sort of a coverage + size metric would be pretty effective at guiding an automated scan such that you'd be able to preserve things liked this without needing humans to go and manually archive each and every page.
AT just asks to be given a few weeks/months of notice depending on how many TBs (MBs?) the crawl needs to be and not to throttle/ban their clients.
I know about the web.archive.org/https://... "hack" but it's probably draining your servers unnecessarily when there are a lot of 301 redirects that weren't archived and on top of that are only able to be validated client side after receiving the whole HTML response.
A REST API would help thirdparty clients to know about this in advance, and the API documentations I found were super unclear in whether something like this exists or not.
Context: I'm building a web browser and I'm trying to offer a feature for error cases when the server or URL isn't available anymore, so that users can see the web archived version of it.
On top of that I have no idea how to "un-UI" the web archived versions. I know that the wget user agent somehow leads to this, but it's also kind of undocumented how the webserver of IA does this in the background and when exactly the UI is injected and all the URLs are rewritten. Something like maybe a http request header to get the raw actual source would be nice.
Have you looked at the headers that they send?
GET /web/20100330210402/https://arxiv.org/abs/0911.1112 HTTP/2
[...]
HTTP/2 200 OK
[...]
link: <http://arxiv.org/abs/0911.1112>; rel="original",
<https://web.archive.org/web/timemap/link/http://arxiv.org/abs/0911.1112>;
rel="timemap"; type="application/link-format",
<https://web.archive.org/web/http://arxiv.org/abs/0911.1112>;
rel="timegate",
<https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>;
rel="first memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT",
<https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>;
rel="memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT",
<https://web.archive.org/web/20110101203756/http://arxiv.org/abs/0911.1112>;
rel="next memento"; datetime="Sat, 01 Jan 2011 20:37:56 GMT",
<https://web.archive.org/web/20211123040625/https://arxiv.org/abs/0911.1112>;
rel="last memento"; datetime="Tue, 23 Nov 2021 04:06:25 GMT"
[...]
The Wayback Machine implements RFC 7089. <http://mementoweb.org/guide/quick-intro/>Brave browser does that, and I've seen Firefox add-ons that do it. Maybe you can look at their code.
I manage the Wayback Machine at the Internet Archive.
Very happy so many people here care about preserving, and making available, our cultural heritage!
Please know a dedicated, and talented, team of engineers works every day to do a better job of archiving more of the public Web, and making it available via the Wayback Machine.
As noted the Internet Archive is experimenting with filecoin.io and storj.io and is always open to suggestions about how we might do our jobs better, and improve our service. We also host regular meetups (and have hosted summits and a camp) related to the Decentralized Web. See: https://blog.archive.org/tag/dweb/
The Internet Archive also offers archive-it.org, a subscription service, for those who want a higher level of support and more features.
We appreciate any support you can offer, financial and otherwise. Please share any bug reports, feature suggestions and other feedback with us via email to [email protected]
Oh/and… checkout the new PDF Search feature we just launched at the bottom of web.archive.org. More to come like that in 2022.
Finally, you might also find some of the things I wrote here of interest: https://gijn.org/2021/05/05/tips-for-using-the-internet-arch...