News for hackers. Stuff that matters.
Re:Waybackpack: download the entire Wayback Machine archive ... by nacs on Tuesday May 03, 2016 @03:52PM (#11623662)
Agreed. From a quick look at code, it seems it just fires off every fetch request immediately after another compeletes.

Hopefully it gets patched to have a built-in rate limit (X requests per minute/hour).

Re:Waybackpack: download the entire Wayback Machine archive ... by derefr on Tuesday May 03, 2016 @07:34PM (#11625074)
If it's already serial, and only works with one backend (rather than being an arbitrary mirroring tool like wget), then the Wayback server can easily "express its preferences" for rate-limiting by adding artificial delay to request-responses that pass the rate-limiting threshold. Backpressure shouldn't be the client's responsibility.

(It only is traditionally, because so many sites do nothing to protect themselves from "being too nice", so arbitrary-backend mirroring-client devs allow their users the option to ask for less than they want. This isn't a sensible protocol design, on either side; it doesn't optimize for, well, anything.)