News for hackers. Stuff that matters.
Re:Official CLI Tool for the Internet Archive by renewiltord on Tuesday October 10, 2023 @04:19PM (#37836925)
Does anyone know how much a text-only mirror of the IA is? I would like to keep such a mirror, but I don't want to store the 57 Petabytes it is right now https://wiki.archiveteam.org/index.php/Internet_Archive#Raw_...

The IA's rate limits are quite aggressive (rightly, I suppose) and so I would like to scan my own text-mirror for the precise point I want and then I can click through to the IA's full wayback machine.

$10/TiB is about what I'd expect for storage drives and perhaps with the right indexing scheme I can spend $10k or less and have a multi-level cache using overlayfs with goofys or some such equivalent.

There are no comments on this comment yet.