Let us serve you, but don't bring us down

Re:Let us serve you, but don't bring us down by yyyk on Monday May 29, 2023 @02:16AM (#36111229)

Archive.org is a bit of a special case, you need to call them repeatedly to archive a website. They do have a rate limit there, it's pretty aggressive* to the point you could trip it by manually using the site. They must have forgotten to limit the OCR files download.

* If they had a better API (a simple non-synchronous API would be enough, one where we could send a list of URLs would be even better), one could have made a lot less calls.

[ Parent | on HN ]

Re:Let us serve you, but don't bring us down by Retr0id on Monday May 29, 2023 @03:42AM (#36111675)

Last time I wanted to bulk-archive a bunch of urls, I asked about it, and sent a txt file full of URLs to someone and they put it in the archival queue.

[ Parent | on HN ]

Re:Let us serve you, but don't bring us down by llui85 on Monday May 29, 2023 @03:48AM (#36111709)

They have a Google Sheets "API" which I've used and works reasonably well:

https://archive.org/services/wayback-gsheets/

[ Parent | on HN ]

Re:Let us serve you, but don't bring us down by krackers on Monday May 29, 2023 @04:50AM (#36112040)

This has been broken for the past month (just stuck on waiting for workers for several days), did they fix it?

[ Parent | on HN ]

Re:Let us serve you, but don't bring us down by abbe98 on Monday May 29, 2023 @09:28AM (#36113729)

I believe you can upload WRAC files to IA and ask them to index the content. Saves them the need to do the archiving and you won't be rate limited on their end.

[ Parent | on HN ]

Re:Let us serve you, but don't bring us down by account42 on Thursday June 01, 2023 @08:57AM (#36150457)

So they just trust you that your archives are not manipulated?

[ Parent | on HN ]