News for hackers. Stuff that matters.
Re:Cloudflare Radar by eminence32 on Wednesday September 30, 2020 @10:41AM (#24639177)
Were these crawlers for big well-known things like google/bing/archive.org? I'm wondering if there are other types of crawlers that I don't even know about.
Re:Cloudflare Radar by drcongo on Wednesday September 30, 2020 @11:11AM (#24639544)
A site for one of my clients was, at one point, getting absolutely hammered by a bot that crawls for plagiarised content. It was ignoring our robots.txt so I ended up blocking it via CloudFlare. There are bots for pretty much anything you can think of out there.

edit: replaced "scrapes" with "crawls" for accuracy.

Re:Cloudflare Radar by oefrha on Wednesday September 30, 2020 @11:01AM (#24639416)
Traders crawl for financial information. Businesses (e.g. airlines) crawl for competitor pricing. Etc. etc.