News for hackers. Stuff that matters.
Re:Internet Archive as a default host-of-record for startups by chaps on Tuesday December 21, 2021 @06:40PM (#29643506)
I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least.

That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin and other needlessly flashy systems. I'm genuinely worried y'all are digging yourselves into a technical pit that you can't get out of and it will hurt or even kill your goals.

Cheers.

Re:Internet Archive as a default host-of-record for startups by smarx007 on Tuesday December 21, 2021 @10:19PM (#29645238)
Recursive scraping:

1) can consume a lot of storage really fast

2) makes IA a bot rather than a user service

3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.

Re:Internet Archive as a default host-of-record for startups by eslaught on Tuesday December 21, 2021 @11:44PM (#29645685)
There's an old school website I wanted to access that's long gone. It's only partially preserved in the Wayback Machine. I'm glad it's there at all, but some of the specific information I was looking for was never captured, and now there's no way to recover it.

I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserving the pages manually, and I don't imagine we'd even show up on ArchiveTeam's radar. But they're not large pages, and basically static. I don't think they'd be a huge burden to store and maintain. It seems like some sort of a coverage + size metric would be pretty effective at guiding an automated scan such that you'd be able to preserve things liked this without needing humans to go and manually archive each and every page.

Re:Internet Archive as a default host-of-record for startups by ivank on Tuesday December 21, 2021 @11:59PM (#29645754)
https://wiki.archiveteam.org/index.php/ArchiveBot is available if you would like to help save things that are still up.
Re:Internet Archive as a default host-of-record for startups by smarx007 on Wednesday December 22, 2021 @05:46AM (#29647655)
I never got turned down sending an IRC message "x is going down in a few days" to the AT channels. https://wiki.archiveteam.org/index.php/Archiveteam:IRC

AT just asks to be given a few weeks/months of notice depending on how many TBs (MBs?) the crawl needs to be and not to throttle/ban their clients.

Re:Internet Archive as a default host-of-record for startups by chaps on Tuesday December 21, 2021 @11:50PM (#29645708)
Pretty much. Your idea of getting coverage could probably work well to make sure IA doesn't "turn into a bot" (though it's practically the same, just without the permanent downloading).
Re:Internet Archive as a default host-of-record for startups by bcherny on Tuesday December 21, 2021 @10:00PM (#29645087)
What’s recursive scraping (presumably, following internal links), and what’s broken about it?
Re:Internet Archive as a default host-of-record for startups by chaps on Wednesday December 22, 2021 @01:13AM (#29646070)
Yep, that's exactly it. Recursive scraping, in theory, would remove a lot gaps on IA. It's really hard to do generically and "right" though. Lots of one-off.. strangeness.. especially from older sites and sites that auto-generate URLs at render time.