News for hackers. Stuff that matters.

A comment on You don’t want to be on Cloudflare’s naughty list. Parent comment

Re:You don’t want to be on Cloudflare’s naughty list by causi on Tuesday September 20, 2022 @12:44PM (#32914342)
What archival tool were you using? I've been looking for a replacement for HTTRACK forever.
[ Parent | on HN ]
Re:You don’t want to be on Cloudflare’s naughty list by NelsonMinar on Tuesday September 20, 2022 @01:42PM (#32915156)
A combination of shotscraper and metascraper; really more web previews than archives. And in a single thread, to different hostnames, maybe one every 10 seconds? Honestly surprised Akamai or anything even noticed. I fake my user agent now, lesson learned.
[ Parent | on HN ]