News for hackers. Stuff that matters.
Re:Internet Archive as a default host-of-record for startups by bcherny on Tuesday December 21, 2021 @10:00PM (#29645087)
What’s recursive scraping (presumably, following internal links), and what’s broken about it?
Re:Internet Archive as a default host-of-record for startups by chaps on Wednesday December 22, 2021 @01:13AM (#29646070)
Yep, that's exactly it. Recursive scraping, in theory, would remove a lot gaps on IA. It's really hard to do generically and "right" though. Lots of one-off.. strangeness.. especially from older sites and sites that auto-generate URLs at render time.