Re:Official CLI Tool for the Internet Archive
by bdcravens on Tuesday October 10, 2023 @03:38PM
(#37836486)
It's a Python script. It's also available via Homebrew, but is curl download any worse than Homebrew or npm install?
(Obviously curl download piped into "sh" or similar is pretty terrible, but at the end of the day, there's a ton of software out there being installed via trust alone)
Re:Official CLI Tool for the Internet Archive
by MenhirMike on Tuesday October 10, 2023 @03:52PM
(#37836635)
It's super useful because on some very large collections with many files, the .torrent doesn't contain all the files (apparently a long standing bug?), and downloading a file that's dozens of gigabytes big through a web browser requires a lot of prayer and faith that the download doesn't interrupt/can be resumed.
So this tool is super handy for that. Heck, it's useful even for small collections just because of how convenient it is.
The IA's rate limits are quite aggressive (rightly, I suppose) and so I would like to scan my own text-mirror for the precise point I want and then I can click through to the IA's full wayback machine.
$10/TiB is about what I'd expect for storage drives and perhaps with the right indexing scheme I can spend $10k or less and have a multi-level cache using overlayfs with goofys or some such equivalent.
Me too, first commit was a bit more than 11 years ago.
(Obviously curl download piped into "sh" or similar is pretty terrible, but at the end of the day, there's a ton of software out there being installed via trust alone)
[1] https://github.com/docsion/rfsh#install
So this tool is super handy for that. Heck, it's useful even for small collections just because of how convenient it is.
The IA's rate limits are quite aggressive (rightly, I suppose) and so I would like to scan my own text-mirror for the precise point I want and then I can click through to the IA's full wayback machine.
$10/TiB is about what I'd expect for storage drives and perhaps with the right indexing scheme I can spend $10k or less and have a multi-level cache using overlayfs with goofys or some such equivalent.